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Abstract 


This thesis examines the design of geographically centralized high performance packet switched 
networks called routing networks. Each of these networks is intended to be used to interconnect the 
modules of a highly parallel computer system. The design of such networks is considered in present 
(1984) technology where only a small number of network nodes can be placed on a single chip and in 
VLSI technology where a large number of nodes can be placed on a chip. 


In both technologies, the design of routing networks for uniform patterns of communication is 
considered. In each technology, it is shown that the characteristics of these patterns imply a minimum 
cost for networks capable of supporting them. In present technology, the performance of a particular 
network that is well suited for uniform communication, the indirect n-cube routing network, is studied. 
The strongest constraint on the performance of the indirect n-cube network that is found still allows the 
the throughput of the network for uniform patterns of communication to grow linearly with the size of 
the network. In VLSI, the use of networks such as the crossbar and the indirect n-cube to support 
uniform patterns of communication is considered. 


The design of routing networks for a few localized patterns of communication is briefly considered 
in both technologies. In cach technology, networks that are well suited for these localized communication 
patterns are discussed. 
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1. Introduction 
1.1 Overview 


Data communication considerations are becoming increasingly important in the design of high 
performance computers. Conventional single sequence computer designs have been refined to the point 
where only limited further improvements in performance can be expected without a dramatic 
improvement in the speed of available circuits. [Levels of performance significantly higher than those of 
present computers can only be achieved by machines capable of substantial concurrency. To achieve 
high concurrency a machine must support a large flow of data and control signals. As we shall see below, 
a class of digital systems which seems well suited for the implementation of such machines in light of their 
communication requirements is the class of packet communication systems [6]. A packet communication 
system is composed of a number of subsystems interconnected by one or more communication networks 
called routing networks. Data transfer between two subsystems is accomplished by passing a packet over 


a network path fiom onc to the other. This thesis examines several aspects of routing network design. 
1.2 Packet Communication Systems 


By definition, any digital system with the following properties is a packet communication system. A 
packet communication system is composed of modules and a set of links where each link connects one or 
more modules. A module can be any form of digital system, and may be capable of storing data and 
performing various operations on that data. The transmission of data from one module to another can 
only be accomplished by passing a packet along a path of connected intermediate modules. Thus the 
behavior of a module can depend only on its internal state and packets it receives from modules 


connected to it. 


Packet communication systems scem well suited for implementing concurrent computers that need 


to support a large flow of data. Consideration of data communication enters at a very carly stage in the 
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design ofa packet communication system. [In particular, the design of the component modules and the 
manner in which they are interconnected should refleet the inherent structure of the computations to be 
performed by the system. By patterning the structure of the system to the structure of the problem the 


designer can develop a system which supports the required flow of data with a minimum of hardware. 


In addition, data communication by packet passing as done in packet communication systems can 
facilitate the efficient use of data paths. This is particularly true for systems such as the Dennis data flow 
machine [7] that require only short messages to be transferred among their componcnt units. Tor these 
machines packet communication has advantages over the alternative circuit switching. In circuit 
switching a complete path between two units must be set up before a message can be transferred from 
one to the other and the entire path must remain allocated for the duration of the transfer. In a packet 
Coiminivaiion Sysicul a imiessage is uansferred in the fonn of a packet that is passed from module to 
module along its desired path. Thus, the message transfer only requires one link at any given time, Only 


the next link in a packet’s desired path need be available for the packet to proceed. 
1.3 Routing Networks 


A routing network is by definition a packet communication system with designated input and 
output links (as shown in Figure 1) that has the ability to accept tagged packets on each of its designated 
inputs and to route cach packet to the output corresponding to its tag. A routing network may be 
connected by its input and output links to other packet communication systems and thus provide 
intercommunication among them. A routing network as a packet communication system is composed of 
packct communication modules that are interconnected by links. For the purpose of discussion, the 


internal modules of a network will be called nodes. 


Routing networks can be used to interconnect several small packet communication systems into a 


larger packet communication system. In such a system, the routing networks handle the required transfer 


Fig. 1. Example Routing Network 


outputs 


of data and control packets among the various subsystems. Thus, an important part of the design of large 


packet communication systems is the design of their routing networks. 


Routing networks are packet networks; data is transferred in the form of packets that are passed 
from node to node in the network. Since routing networks are also packet communication systems and 
since the control of a packet communication system can not be centralized, the routing of packets must be 
done in a decentralized fashion. The routing decisions made by a node for a given packet can depend 
only on the state information of the node and the labcl of the packet. There are, however, a number of 
differences between these networks which arc intended for usc in localized high performance systems and 
distributed packet networks such as the ARPA network. In contrast to the ARPA network where the cost 
of data paths dominates, both the cost of data paths and the cost of nodes must be considered in the 
design of a routing network. Thus, in the design of a routing network it is important to minimize the 
complexity of network nodes. Very large buffers for queuing packets or very large tables for storing 
routing information cannot be used. Unlike the ARPA nctwork, a routing network has data paths and 
nodes of comparable speed and reliability. ‘his suggests that a simple routing algorithm should be used 


by cach node in a routing network in order to minimize the total time required for a packet to pass 


through the network. 


Routing networks differ from the majority of networks that have been studied in classical switching 
theory. While routing networks use packet switching, switching theory has been primarily concerned 
with networks that use circuit switching. Further, in contrast to routing networks that use decentralized 
control, most of the networks that have been studied in switching theory use centralized control. Finally, 
switching theory has assumed that the cost of wires is negligible in comparison to the cost of active switch 
elements. This assumption, as we sce below, is not valid for some of the technologics that may be used to 
implement routing networks. Thus, while classical switching theory provides a good starting point for 
rescarch on routing networks, most of the results that have been obtained do not directly apply to routing 
networks. In this thesis, we examine cost and performance issues for routing networks that are similar to 
the cost and perforiiiaice issues that lave veer studied for circuit switched newworks i Classical switching 
theory. We will use cost measures that are appropriate for present and future integrated circuit 


technologies, and performance measures that are appropriate for the intended network applications. 
1.4 Research Topics 


This thesis examines the design of routing networks under two different sets of assumptions that 
correspond to two points on the apparent path of integrated circuit technology evolution. One sect of 
assumptions corresponds to present technology where only a small number of nctwork nodes can be 
implemented on a single integrated circuit. The other set of assumptions corresponds to a technology 


where a large number of network nodes can be implemented on a single integrated circuit. 


In 1984 technology, consideration of the length of wire needed to implement a given link seems 
unimportant, It scems unlikely that more than a sinall number of modules can be implemented on a 
single integrated circuit. Links between modules can be implemented for the most part as printed circuit 


board wires. The length (as opposed to the number) of such wires is not a significant factor in the total 


costof the system, Similarly, the effect of wire length on system speed is small since even the propagation 


time of a long wire is less chan the delay of the circuit required to drive it from an on-chip signal. 


In Chapter 2, we examine the design of routing networks for present technology. We assume that 
the length of wire required to implement each network link does not affect cither the cost or the 
performance of the network. Further, we place certain additional restrictions on the behavior and 
complexity of network nodes in order to narrow down the nuinber of design parameters that must be 
considered. These restrictions, which are described in Section 2.2, are motivated by the current state of 


technology and the nature of the systems in which the networks may be used. 


Given these restrictions, we examine in Section 2.3 the design of networks for a class of systems 
characterized by uniform communication; cach source of packets in such a system gencrates packets for 
all the possible destinations and over the long run generates a comparable number for each destination. 
For the purpose of analysis, we introduce simple probabilistic models of the packet sources and sinks of a 
system with uniform communication. We examine the minimum number of nodes required by any 
network that is capable of high throughput when it is connected to the model sources and sinks. We 
study a particular network, the indirect n-cube routing network, that seems well suited for uniform 
communication and has a number of nodes within a constant factor of the lower bound. Below, we use 


the term indirect n-cube network to refer to the indirect n-cube routing network. 


It should be noted that networks related to routing networks in general and the indirect n-cube 
network in particular have been studied in the literature. Sorting networks, networks capable of sorting 
N data items in parallel where N is the nuinber of network inputs, are clearly not the same as routing 
networks, but intuition would suggest that these two types of networks have similar complexitics. Sorting 
networks using O(N log2 N) nodes have been known for some time [4]. More recently, O(N log N) node 
sorting networks have been described [3], although these networks may not be of practical interest 


because of the very large constant factor. ‘The indirect n-cube network has a comparable complexity: it 
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uses O(N log N) nodes. Earlier work on routing networks with structures similar to that of the indirect 
n-cube network has been done [5]. Other networks with related structures have been described in the 
literature. Some of these networks have been proposed to perform permutations on large vectors of data. 
In general, these permutation networks have been proposed for systems in which the elements of a given 
vector are processed synchronously and then permuted. These permutation networks include 
shuffle-exchange, omega, Pease’s indirect n-cube permutation network, and cube-connected cycles [24, 
14, 15, 16, 21, 23]. Other networks with related structures have been proposed to interconnect the 
processors and memories of other types of multiple processor systems. Circuit switched and packet 
switched banyan networks have been proposed for single instruction stream multiple processor systems 
[10, 26]. Circuit switched and packet switched delta networks have been proposed for multiprocessors in 
which each processor makes independent and random memory accesses [20, 8]. The relationship between 
the indirect n-cube routing network and previously studied networks is discussed in more detail in 


Section 2.2.3.1. 


We consider in Section 2.3 the operation of large indirect n-cube routing networks when connected 
to the model sources and sinks and examine certain important characteristics of the operation of these 
networks. We examine the influence of these characteristics by using network modcls that accurately 
model these characteristics and that are considerably simpler than the actual network. By analyzing and 
simulating the models, we examine the influence of these characteristics on certain aspects of the 
performance of the networks. ‘The performance predicted by these models is compared to the 


performance of the actual network which we measure by simulation. 


One important aspect of performance that we study is throughput. We examine the relationship 
between network throughput and network size. We would like the throughput of the nctwork to scale 
lincarly with the number of network inputs since if we form a composite packet communication system 


by using a routing network to interconnect several subsystems, we would like the performance of the 
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system to scale linearly with the number of subsystems. 


Another important aspect of performance that we study is the speed of slow inputs. [fa particular 
network input becomes extremely slow due to network congestion then packets from the module 
connected to that input can be delayed. If the congestion continues for a long period, a large number of 
modules that are cither directly or indirectly dependent on the blocked module in a highly parallel 


computation can be affected. 


Our study suggests that very large indirect n-cube networks can support high performance for 
uniform communication patterns. The strongest constraint on network throughput that we find in our 
study still allows throughput to grow linearly with network size. However, our study also indicates that 


some of the inputs of a very large network can be slow for a very long period of time. 


We also briefly examine in Section 2.3 the design of routing networks for a class of systems 
characterized by localized communication; the majority of packets generated by a particular source in 
such a system are taggcd for a small group of destinations. Many localized communication patterns can 
be supported with networks that are less complex than the indirect n-cube network. We discuss one 
obvious family of networks that seem appropriate for some important localized communication patterns. 
One of the characteristics of networks of this family is a number of nodes equal to the sum of the number 
of network inputs and the number of network outputs. This family includes grid structured networks and 


tree structured networks. 


In Chapter 3, we examine the design of routing networks in the technology of five to ten years from 


now. 


As technology changes and the number of network nodes that can be placed on a given integrated 


circuit increases, the importance of wire length in nctwork cost will increase. Within a few years it should 


be possible to implement many modules and the links that interconnect them on a single chip. ‘The 
length of wire required to implement cach link will then be a significant factor in the cost of silicon 
required to implement a system, since the chip acreage used by cach wire is proportional to its length. 
However, it is likely that for the immediate future, the next five to ten years, even long wires can be 


driven quickly if drivers of the appropriate size are used. 


In Chapter 3, we make some assumptions about the characteristics of the integrated circuit 
technology of the next five to ten years and study the design of routing networks under these 
assumptions. For the purpose of discussion, we refer to the technology that will exist at the end of this 
period as very large scale integration (VLSI), We describe a model of VLSI based on some assumptions 
about the characteristics of VLSL We examine in the VLSI model the fundamental cost of a single chip 
iiviworn to Suppurt a certain level of perfurtnance for unifuan paiierns of communication. We examine a 
few structures that seem appropriate for implementing a single chip uniform communication network in 
VLSI. These structures include a crossbar structure, and an indirect n-cube structure. We discuss a 
technique for interconnecting such single chip networks to form larger uniform communication networks. 
We also bricfly examine the design in VLSI of networks for localized patterns of communication. We 


examine a few example network structures and describe the communication patterns that they can 


support. 
1.5 Notation for Asymptotic Bounds 


We usc the following notation to describe asymptotic bounds. 


We say that F(N) is Q(g(N)) if and only if there exists Ng and c greater than zero such that f(N) is 


greater than or equal to ¢ g(N) for all N greater than No. 


We say that FCN) is O(g(N)) if and only if there exists No; Cy. and Cy greater than zero such that 


cy 8(N) < c7R(N) for all N greater than No: 


We say that f(N) is O(g(N)) if and only if there exists Ng and c greater than zero such that f(N) is 


less than or cqual to c g(N) for all N greater than No: 


2. Design of Routing Networks Ignoring Wire Length 
2.1 Network Restrictions 


In general, we will be concerned only with routing networks that obcy the restrictions described in 


the following paragraphs, 


We assume that time is divided into units and that any operation in the network starts on a 
transition between two time units and finishes before the next transition. [t is important to note in 
passing that this assumption of purcly synchronous behavior is only for the convenience of analysis and 


that the networks that we shall present can easily be designed to function asynchronously. 


We assume that there is some limit & on the total number of input and output links that can be 
attached to a particular network node, and we assume that there is a limit o on the total number of 
packcts that may be buffered at any particular nctwork node at any particular time. hese restrictions are 
motivated by our desire to bound the amount of chip space and number of external connections required 


to implement a network node as a portion of an integrated circuit. 


Tor the most part, we assume that only two nodes are connected to a given link. The only exception 
is our discussion of the minimum cost of a network to support high throughput for uniform patterns of 
communication. In that discussion, we get a more gencral lower bound by assuming that an arbitrary 
number of nodes can be connected to a given link. [n all cases, we assume that one link can transfer at 


most one packct per unit time. 


In the case that only two nodes are connected by a link, we assume that the two nodes observe a 
ready/acknowledge protocol for transferring packets. In particular, each link contains acknowleze and 
ready control lines in addition to the data lines used to transfer the packets as shown in Figure 2. ‘The 


protocol has 4 phases as shown in Figure 3. A sending node can place a new packet on the dara lines of 


Fig. 2. Lines of a Link 
data 
ready 


acknowledge «0 -----— 0 --2 eee - 


Fig. 3. 4 Phase Protocol 
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the link only if all 4 phases of the transmission of the previous packet have been completed. The sending 
node asserts the ready line after placing the packct on the data lines and will continue to assert the ready 
and data lines until the acknowlege line is asserted by the receiving node. The receiving node can accept a 
packet available on the data lines only after the ready line has been asserted, and the receiving node will 
assert the acknowlege line only after it has safely stored the packet in one of its buffers. This protocol was 
chosen since it is widely known and provides in a straightforward manner the necessary coordination for 


packct passing. 


We assume that the behavior of a node during a given time unit can depend only on the 
information available on its links and the contents of packets stored in its buffers, and we assume that the 
node's behavior can depend only on the destination label and not the data portion of any packet stored in 
the node. These restrictions are in part motivated by our desire to implement cach network node as a 
portion of an integrated circuit. By limiting the complexity of cach node's behavior we limit the amount 


of space required to implement the control circuits of each node. These restrictions are also motivated by 


our desire to minimize the time required for a packet to traverse the network since by limiting the 
complexity of the control algorithm of each node we indirectly limit the time required by each node to 
process a packet. ‘These restrictions seem natural and can be casily observed in the design of networks. 
However, it should be noted that there are many alternative sets of restrictions that could be placed on the 


node’s behavior, and that a different sct of restrictions might well lead to different network structures. 


Finally, we assume that a packet that enters a node in a given unit of time can not leave that node 
until the next unit of time, and that a link can transfer at most one packet per unit time. Although we are 
ignoring the time required to propagate a packet over a link, we are not ignoring the time required to gate 


a packet through a node or store it in a buffer. 
2.2 Networks for Systems with Uniform Communication 
2.2.1 Model of the Problem 


In general, a routing network will be used as shown in Figure 4 to connect a group of packet sources 


to a group of packet receivers. Each source will produce labeled packets and each packet must be 


Fig. 4. Use of a Routing Network 
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delivered to the reeeiver which corresponds to its label. 


We describe here a simple model of a system with uniform communication. For our model, we 
assume that the number of receivers equals the number of sources and that reccivers and sources behave 
in a very simple manner. We assume that a recciver will process a message as soon as it arrives and can 
thus accept a packet every unit of time. We assume that if the network has accepted the last packet 
generated by a particular source, then the chance that the source will produce a new packet in a unit of 
time is P where P is a parameter of the model. We assume that the label of cach packet is independently 


sclected, and that all of the possible receiver labels are equally likely. 
2.2.2, Minimum Network Cost 


in this section, we tind a lower bound on the complexity of any N-input N-output routing network 
capable of Q(N) average throughput when each of its inputs is connected to a model source and each of 
its Outputs is connected to a model receiver. We measure throughput as packets per unit time and 
complexity as the number of nodes in the network. We show that such a network requires Q(N log N) 
nodes. This result gives some motivation for the fact that the network that we study for such applications, 


the indirect n-cube network, and most related networks require Q(log N) stages of Q(N) nodes. 


For the purpose of this discussion, we allow an arbitrary number of nodes to be connected to a 
given link. As before, we allow only one packet to be transferred over a given link in a given unit of time. 
Clearly, the lower bound on network cost that we obtain by allowing an arbitrary number of nodes to be 


connccted to a given link also holds if only two nodes are allowed to be connected to a given link, 


Proposition. Q(N log N) nodes are required by any N input N-output routing network capable of 
Q(N) average throughput when each of its inputs is connected to a model source and each of its outputs is 


connected to a model receiver. 


Proof, Since we assume that a network node can be connected to at most & links, we can get a lower 
bound on the number of network nodes by finding a lower bound on the sum, over all links in the 
network, of the number of nodes connected to each link. For the purpose of this discussion, we use the 
term connection to refer to the juncture between a network link and a network node. Thus, the sum, over 
all links in the network, of the number of nodes connected to each link is equal to the number of 
connections in the network. We get a lower bound on the number of connections in the network by 
considering the sum, over all the packets processed during a‘long period, of the number of connections 
uscd GY ach packet aNd LY Making use uf the fact that a connection can be involved in ac most one 


operation per unit time. 


A lower bound on the number of connections in the network can be obtained by considering the 
operation of the network over a long period of time and examining the use of network connections during 
such a period. Let us consider a period of T time units for some very large T. Since we assume that only 
one packet can be transferred on a link in one unit of time, it follows that a connection can be used for 
only one packet in a given unit of time. The total number of connections in the network must be at least 
as great as (1/T) times the number of connection operations during the period where the number of 
connection operations is defined to be the sum, over all packets processed, of the number of connections 
used by each packet. It should be noted that we count each connection of a link separately, and that the 


the number of connections used by a packet includes each connection of each link used by the packet. 


A lower bound on the expected number of connection operations during the T time unit period can 
be obtained by considering the possible paths through the network. For the purpose of this discussion, 


we assume that each network fink is logically composed of some number of link segments. In particular, 


we assume that the nodes connected to a given link are connected according to some linear order, and 
that a link segment is a portion ofa link between two adjacent nodes as shown in Figure 5, A link which 
is connected to ¢ nodes has i - 1 segments. By this definition, a connection between a link and a node 
can involve at most two link segments, and at most 2 & link segments in total can be connected to a node. 
Less than 

: ayitl. 

(2k-1)9 + Qk-D! +“ k-1)! = ST a (1) 
outputs can be reached from a particular input by a path containing jor fewer link segments. Thus, there 
are less than N/2 outputs that can be reached from a given input using a path containing no more than 
(log(a x -1) (N/2)) - 1 link segments. Since a model source randomly selects a destination for each packet 


it generates and since all destinations are equally likely, there is at least a 50% chance that a packet 


generated by a model source must travel over a path of greater than (log(y 4 5 (N/2)) - 1 link segments. 


Fig. 5. Conceptual Model of Nodes Connected to a Link 
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It follows that the expected number of link segments used by such a packet must be Q(log N). Since the 
number of nodes connected to a given link is equal to the number of segments in the link plus one, the 
total number of nodes connected to the links in the path of a particular packet must be larger than the 
number of link segments in the path. ‘Thus, a packet generated by a model source is expected to use at 
least O(log N) connections. Since by assumption the expected number of packets processed in the T time 
unit period is QCLN), the expected number of connection operations during the period must be greater 


than Q(1'N log N). 


A lower bound on the number of nodes in the network follows. Since the expected number of 
connection operations during the T time unit period is Q(TN log N) and since a connection can be 
involved in at most one operation in a given unit of time, the network must have Q(N log N) separate 
comMections Ociwecn uixs and Nudes. Since each tude cau be connected to ai musi A Hiks and since & is 


fixed and is independent of N, the network must have Q(N log N) nodes. 


Thus, there must be Q(N log N) nodes in a N-input N-output routing network capable of Q(N) 
average throughput when cach of its inputs is connected to a model source and cach of its outputs is 


connected to a model receiver. 

This ends the discussion of the proposition. If 
2.2.3 Routing Networks Using an Indirect n-Cube Topology 
2.2.3.1 Introduction 


The network shown in Figure 6, the indirect n-cube network, seems well suited for applications 
with uniform communication and has a cost of the same order as the lower bound derived in the previous 


section. 
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In this introduction, we describe the indirect n-cube network, we discuss Me relationship between 
previously studied networks and the indirect n-cube network, and we give a brief overview of our work 


on the indirect n-cube network. 


An indirect n-cube network is constructed as shown in Figure 6. A N-input network is composed 
from two N/2-input networks and N/2 nodes (called routers). Each node has two input and two output 
links. This construction yields an interconnection which is topologically equivalent to the interconnection 
of butterflies in the radix two fast Fourier transform [11]. In total (N/2) logy N nodes are required. One 
and only one path exists from a given input to a given output. If network outputs, stages, and node 
outputs are numbered as shown in Figure 7, then at the ith stage the appropriate path follows the node 


output that corresponds to the ith most significant bit of the binary representation of the number of the 


5 et poems Seem eee Se 
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Fach node of the network can be structured as shown in Figure 8. The node has a fifo buffer 


capable of storing some number of packets on each of its input links. If at the beginning of a time unit a 


Fig. 6. NxN Indirect n-Cube Network Construction 
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Fig. 7. Example Indirect n-Cube Network 
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buffer is not full and the corresponding input ready line is asserted then the node control places the 
packet available on the daéa lines in the buffer and asserts the returning acknowlege line before the end of 
the time unit. Ifa buffer is not empty at the beginning of a time unit then the node control attempts to 
place the packet which entered the buffer first on the output link corresponding to its destination. If the 


node control can do this, cither because no conflict exists or because of arbitration of the conflict, then it 
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asserts the corresponding output ready line. [fan acknowledge is returned on that link before the end of 
the time unit, then the node control removes the ready and removes the packet from the buffer. 


Otherwise, ready and the packet are removed during the first time unit when acknowledge is returned. 


As was mentioned in the first chapter, there are a large number of networks that have structures that 
are related to that of the indirect n-cube network. ‘The topology of the indirect n-cube routing network is 
identical to that of several other networks including omega, s=f=2 banyan, and Peasce’s indirect n-cube 
permutation network [16, 10, 21]. Networks with the same topology have also been called delta networks 
[20]. The topology of the shuffle-exchange network [24] is also related since an omega network is simply a 
cascade of logy N shuffle-exchange networks. As was mentioned in the first chapter, these related 
networks have been proposed for a variety of uses. Some ‘of these networks have been proposed to 
pcrtorm permutations on large veciuis of daia. In general, these permutation networks have been 
proposed for systems in which the elements of a given vector are processed synchronously and then 
permuted. Other related networks have been proposed to interconnect the processors and memories of 


other types of multiple processor systems. 


The work on networks for the synchronous permutation of large vectors of data has becn mostly 
concerned with the types of permutations that can be realized by a given number of passes through such a 
network, and thus that work docs not directly address the question of how well such networks perform 


when interconnecting the modules of a packet communication system. 


Some of the work on interconnection networks for other types of multiple processor systems is 
more closcly related to our study of the indirect n-cube network. We discuss a few picces of this work. 
The first is the work of Valiant [28]. Valiant has suggested the use of networks such as the packet 
switched n-cube for interconnection of processors and memories in a synchronous multiprocessor system. 
Valiant introduces the concept of an idealistic computer composed of processors that operate 


synchronously and a memory that the processors share. He considers algorithms such that no memory 
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location is accessed in a given computational step by more than some constant number of processors. He 
assumes that the idealistic computer can implement each computational step in a single unit of dme. He 
considers the simulation of the idealistic computer on a realistic computer with a packet switched n-cube 
network, Each node of the network has the capacity to buffer a number of packets proportional to the log 
of the number of processors. Valiant shows that with high probability the realistic computer can 
implement a computational step in time proportional to the log of the number of processors. While it 
seems plausible that the memory accesses corresponding to several computational steps can be pipelined 
in the realistic computer, Valiant does not show this. Thus, Valiant’s work differs from ours in at least 
three ways. First, he considers systems of synchronous processors and we consider systems of largely 
independent asynchronous processors. Second, in each network nodc he allows buffering proportional to 
the log of the number of processors and we allow only buffering of fixed size. Finally, he does not 
consider the pipelining of packcts of different computational steps through his network and we consider a 


continual flow of packets through our networks. 


Upfal (27] has shown similar results for networks of fixed degree. He uses the d-way digit-exchange 
graph. A processor is associated with cach network node and each network node is assumed to have 
O(log N) buffers where N is the number of processors. It is assumed that a packet is initially at each 
processor and that cach packct is destined for some other processor. No two packets are destined for the 
same processor. Upfal shows that with high probability all the packets can be delivered to their 


destinations in O(log N) time. 


Patel has suggested the use of circuit switched delta networks for multiprocessors in which cach 
processor makes independent and random memory accesses [20]. For his analysis, Patel assumes that 
memory requests in a multiprocessor are generated in a manner similar to the manner in which packets 
are gencrated by our model sources. The primary distinction between our work and that of Patel is the 


fact that our routing networks are packet switched. In Patel’s circuit switched network, the transmission 
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of a message requires the use of a circuit through all of the stages of the network. In our routing 
networks, at any given time a packet only uses one link to go from its present stage to the next stage. ‘The 
throughput of Patel’s networks do not grow linearly with their size. As we shall sce, there is reason to 
believe that the throughput of the indirect n-cube network for uniform patterns of communication docs 


grow linearly with its size. 


Dias and Jump [8] have done work on the use of packet switched delta networks as interconnection 
networks for multiprocessors and this work is very closcly related to our study of the indirect n-cube 
network, Their network is topologically identical to the indirect n-cube. Their analysis assumes that the 
packets and the labels on the packets are generated in a manner similar to the way that packets and packet 
labels are gencrated by our model sources. They analyze their networks using network models in a 
anes siiiilar to the way that we aiialyze ihe iudirecit a-cube network using network models, However, 
their models differ from ours. They use a Markov model to develop approximate equations for the state 
probabilitics of a router in a given stage in terms of the state probabilitics of the routers connected to it. 
They simultaneously solve the equations for all the stages. Their analysis makes several approximations. 
The analysis assumes that the routers of a given stage are independent. Also, the analysis of a given 
router assumes that the state probabilities of routers connected to it are independent of the state and 
history of the given router. Some of the characteristics of network behavior that we study in our models 
violate these assumptions of independence. For modest sized networks, the nctwork throughput 
predicted by our models is consistent with that predicted by their models. However, for very large 
networks our throughput predictions differ from theirs. Since their modcl assumes more independence 
than ours one might expect it to predict higher throughput, but in fact the way that their assumptions are 
used in their model leads to a prediction of lower throughput. Their model predicts that the normalized 
throughput goes to zero as the network size goes to infinity [9] where normalized throughput is defined to 
be network throughput divided by network size. All of the constraints on network throughput that we 


study allow a non zero asymptote. We bclicve that our study considers all of the constraints represented 
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by their model and several that are not. We believe that the asymptotic prediction of their model is the 
result of the way that their assumptions are used in their model and does not reflect any real constraint on 
the throughput of the network. Another difference between their work and ours comes from the fact that 
in addition to studying the average network throughput we also consider the speed of slow network 


inputs. Their work is primarily concerned with the average network throughput and delay. 


Recently, Pippenger [22] has extended the results of Valiant and Upfal to a network with a fixed 
amount of buffering at cach node. In his work, Pippenger uses the d-way digit-exchange graph. A 
processor is associated with each network node. Only a fixed amount of buffering is assumed at cach 
node. [t is assumed that a packet is initially at cach processor and that cach packet is destined for some 
other processor. No two packets are destined for the same processor. Pippenger assumes that each node 
OUCYS Cefiaiis FUles CUiicerning the order in whit ti processes the packeis thai it receives, aud siiaws Ulat 
if the rules are obeyed then with high probability all the packets can be delivered to their destinations in 


O(log N) time. 


While there are differences between Pippenger's work and ours, Pippenger’s results are significant 
and have some bearing on our work. Pippenger’s network differs from the indirect n-cube network; the 
indirect n-cube network would be more closely related to Pippenger’s network if the inputs of the indirect 
n-cube network were connected to the outputs and if a processor were associated with cach network node 
of the indirect n-cube network. The type of network operation that Pippenger considers differs from the 
type that we consider; Pippenger docs not consider the pipelining of waves of packets through his 
network and we consider a continual flow of packets through the indirect n-cube network. However, by 
establishing certain additional rules for the operation of the nodes of the indirect n-cube it may be 
possible to extend Pippenger’s approach to provide results on the performance of the indirect n-cube 
network for uniform communication, ‘The additional rules would concern the order in which a network 


node processes the packets that it receives, and possibly the removal of unusual blockages. It may be 


possible to show that the normalized throughput of the indirect n-cube network with the additional rules 
approaches a non zero constant. Such a result is plausible, but even if the result holds it may be very 
difficult to prove. In any case, such a result is consistent with our work. Our work considers the 
performance of the indirect n-cube without rules such as those mentioned above. Even without such 
rules, the strongest constraint on network throughput that we find still allows normalized throughput to 


approach a non zero constant. 


In the following pages, we examine the effect of certain important characteristics of the operation of 
very large indirect n-cube networks. In particular, we study in 2.2.3.2 the effect of congestion at a single 
router, In 2.2.3.3, we study the effect of congestion in a single stage of routers. In 2.2.3.4, we study the 
effect of the interaction of routers of different stages. As was mentioned earlier, we examine the effect of 
uicse Characteristics of etwork Uchavior by using network models thai accurately model these 
characteristics and that are considerably simpler than the actual network. By analyzing and simulating 
the models, we examine the effect of these characteristics on network throughput and the speed of slow 
network inputs. As was mentioned earlier, our study suggests that very large indirect n-cube networks 
can support high performance for uniform communication patterns. The strongest constraint on network 
throughput that we find in our study is caused by the interaction of routers of different stages and it still 
allows throughput to grow linearly with network size. However, our study of the interaction of routers of 
different stages also indicates that some of the inputs ofa very large network can be slow for a very long 


period of time. 
2.2.3.2 Tree Buffering 


The first characteristic of nctwork operation that we examine is tree buffering. We use the term tree 
buffering to refer to the buffering of packets that occurs in front of a congested router. Such buffering 
involves a tree structure of buffers. As a result, congestion at one router in a given stage can affect a large 


number of other routers. 
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In this section, we examine tree buffering in front of a single congested router. In the next section, 
we cxamine a stage of routers and we examine the probability that at least one of the routers of the stage 


has a deep tree buffered in front of it. 


In this section, we use a network model to examine how much tree buffering can occur in front of a 
congested router. In the model congestion can occur only at one router. We will study the model in 
order to determine the amount of buffering that occurs as a function of the amount of congestion and the 


overall network input rate. 


As we shall sce, the model suggests that deep tree buffering in front of a slow router occurs if the 
rate at which the router can accept packets is close to the rate at which packcts that must go through that 
router are generated. In particular, the model suggests that if the rate at which packets are generated on 
each network input is /N and if the rate at which the router can accept packets on its input is OUT’ then 
the expected number of packets buffered in front of that input is greater than 


IN 
(SmnTI UPR Th 7 2 
2 20UT-IN \(B +1) 2 1 where B is the size of the buffers. It should be noted that some aspects of 


IN IN/OUT 


this expression are intuitive. The OUT-IN © T-IN/OUT factor in the exponent is similar to the 


expected occupancy of certain types of queues in classical queueing theory, and reflects the queucing of 
packcts that must be passed through the congested router. ‘The exponential growth in the total number of 


packets buffered, most of which do not have to pass through the congested router, comes from the tree 


structure of routers involved. The 3 L i factor in the exponent reflects the influence of the size of the 


input buffers of the routers. 


The model that we use is shown in Figure 9. The model is composed of a tree of routers as shown 
in the figure. The depth of the tree is a parameter of the model. The first output of the router at the root 
of the tree is connected to a probabilistic packet sink and the second output is connected to a perfect 
packet sink. All other routers have their first output connected to a router in the following stage and have 


their second output connected to a perfect packet sink as shown in the diagram. Each input of cach 
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Vig. 9. Tree Buffering Model 
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router at a leaf of the tree is connected to a probabilistic packet source. 


The tree of routers of the model represents the tree of routers in front of a congested router in the 
network. The probabilistic sink represents the congested router. The perfect sinks represent the routers 


directly connected in the network to the tree of routers being studied. 


The routers in this model, unlike the routers of the network, operate instantly. ‘Thus, a packet will 


ripple through the model in one time unit. It will cither be output ata perfeet sink or it will run into 
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other packets buffered as a result of congestion at dhe probabilistic sink. ‘This assumption allows us to 
easily study buffering due to congestion at the probabilistic sink since it is the only type of buffering that 


OCCUTS, 


The fifo buffers in the routers all have the same size B. B is a parameter of the modcl. 


The probabilistic packet sink contains a fifo buffer whose size is the same as that of the buffers in 
the routers. Packets input to the sink are placed in the fifo buffer. If at the beginning of a time unit the 
fifo is not empty then with probability OUT the sink removes one packet from the fifo buffer. OUT isa 


parameter of the model. 


The perfect sinks never block and accept packets at whatever rate they are presented. 


The probabilistic packet sources produce packets. If the input buffer connected to a probabilistic 
source is not full at the beginning of a time unit, then with probability 7N the probabilistic source places 
an additional packet in the buffer. /N is a parameter of the model. The tag for cach packet has as many 
bits as the depth of the network. Each bit of each tag is independently and randomly sclected with one 


and zero being equally likely. 


We can obtain a rough understanding of the operation of the model without much effort by 
considering the packets that are buffered in the model and that are tagged for the probabilistic sink, and 
by examining the expected number of such packets as a function of JV and OUT. For the purpose of 


discussion, we call these bps (buffered probabilistic sink) packets. 


bps packets can only leave the model at the probabilistic sink. From the operation of the model we 
can conclude that if at least one bps packet exists then the buffer in the probabilistic sink must contain at 
least one packet. ‘Thus, in a given unit of time if any bps packets exist then with probability OUT one will 


be consumed by the probabilistic sink. 
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A bps packet can enter the network from any one of the probabilistic sources. Hf the depth of the 
model is d then the number of sources is N where N is equal to 24 The chance that an unblocked source 
produces a bps packet in given unit of time is /N/N. If N is large and if none of the sources is blocked 
then the chance that k bps packets are produced in a given unit of time may be approximated by 


k IN 
UN) a : ae: : : : : 
E é . The accuracy of this approximation increases with the size of N and is exact for infinite N. 


In the following paragraphs we will examine for a very large network model the expected number 
of bps packets as a function of IN and OUT. We will assume that OUT > IN. We will assume that the 
network is large enough that the chance of a blocked source is small and can be ignored. We will assume 


that the network starts at tine zero with no bps packets. 


Proposition. The expected value of the limiting distribution for the number of bps packets is equal to 


2IN-IN2 2) 
XOUT-IN)* 


Proof, We find the average number of bps packets using an approach similar to that used by Kleinrock 
for the M/G/1 queue [12]. For the purpose of discussion, we usc the notation g,, to represent the 
number of bps packets at time n. We use Dopey to represent the number of bps packets served between 
nand va+l. Boa is of course equal to cither zero or one. We use Vy 41 (0 represent the number of bps 


packets generated between a and n+1. 


From these definitions it follows that q, , ,, the number of dps packets at time n-+ 1, is given by the 
equation 
Int1 = % “Anti t N41: GB) 
If we square both sides we get 
Dg Be Ree Ge Sg Ne Og pe ON ey (4) 
Int. = In ntl n+l 4nOn+l Gn n+l nt Vat: 
Let us form the expectation of both sides. We use the notation E[x] to represent the expected value of 


x. Also we make use of the fact that Doe i the square of the number of bps packets served between # 


and a+ 1. is either one or zero and is equal to Ae a: I: Edy 4 a the expected value of the square of the 
number of bps packets at time #+ 1, is given by the equation 
Edy g T= Ely 1+ EO + Pb I 
Eg, Ay galt Fld gl FA ag (5) 
Since Vy 4 p the number of bps packets generated between n and #+ 1, is independent ofq, and aane 1 
Edy 417) = Elgg 1+ ETA gg t+ Dag 7) 


Edy Ang l + 2F Tay ME ly gh 2FTA gy VED 4 - (6) 


We arc interested in the limit as 1 goes to infinity. We are interested in the limiting distribution for 

the random variable q,,, the number of bps packets at time n. We denote the limiting distribution by q. 
Similarly we use the limiting distribution for the. random variable v, the number of bps packets 
generated between n and n+l. We denote the limiting distribution by ¥. We assume that the /th 
moment of gq, exists in the limit as # goes to infinity independent of , namely, 

lim 500 FUdy/1 = HIG). (7) 
We make a similar assumption about the jth moment of y,. We make use of the fact that lim, _, 99 
ELA, ,] must equal the average input rate, /V. Thus, 

EUG?) = EWG?) + IN + ELV] + 2ELZ EL - 2UN ELF] - tim, og 2ELay, Ay 1B) 


The probability that raven = I given that q, >0is OUT. Thus, 


lim, 509 F(a, Ang] = (OUTIELT| (9) 
and 

O0=IN + E [#2] + 2E[GJE[V]- 2UN)E[¥]- 2Q00UT)E [GZ]. (10) 
The probability that ¥ = & is cies This of course is the Poisson distribution. Thus, 

E(V]= IN, (11) 

E[32] = IN2 + IN , (12) 


and 


O= IN + IN2 4 IN + UIN)E[G]-20N2- AOUT ETT] (13) 
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or 
QIN - IN2 = XOUT = INJE|G]. (14) 


Therefore, /[q], the expected value of the limiting distribution for the number of bps packets, is equal to 


NTN 2 
2IN-IN (15) 
2OUT-IN ) 


This ends the discussion of the proposition. ff 


Thus, the expected number of dps packets is very large if and only if OUT is very close to 7N. For 
example, if OUT -IN is equal to 1/a for some constant a then the expected number of bps packets is less 


thanalN. 


Other measures of the amount of buffering in the network model can be deduced from this result 


for bps packets. 


For the purpose of discussion, we define some notation. We use the notation Py tO denote the total 
number of packets buffered at time ». We use p to represent the limiting distribution for the random 
variable p,,. We define f; (i) as follows. 

f, (i) = 0, ifi=0, 
and 
Sf, (i) = logy i, if DO. 
(16) 


We use /,, to represent f) (DP, ). We use? to represent the limiting distribution for the random variable 
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Proposition, ‘The expected value of p, the limiting distribution for the total number of packets 
buffered, is greater than 

{XOUT-INKB +) 7). (17) 
Proof. ‘Vhe desired result can be obtained from a lower bound on the expected value of 7, the limiting 
distribution for f;(p, ). And EG] in turn can be obtained from the expected value of 7, the limiting 


distribution for the number of bps packets. 


I,» £7 @,), can be related to qg,, the number of bps packets at time n, We use P[x = i] to 
represent the probability that x equals i and P[x = ily = /] to represent the probability that x equals i 
given that y equals j. Clearly, £ (4, ], the expected number of dps packets at time a, is equal to 


Eioo Zyp0 JPlay = ily ~OPL AH OL. (18) 
We use the notation £' [x |y = i] to represent the expected value of x given that y equals i. Thus, 


Eldyl = 09 (Elay ly =f ODPL, =H OI. (19) 


An upper bound on £'[q,, ln = f, (i)], the expected number of dps packets at time n given that the 
total number of packets buffered at time # is equal to 7, can be obtained by examining the model in more 


detail. 


Packets buffered in stages close to the probabilistic sink are more likely to be bps packets than 
packets buffered in distant stages. Consider Figure 10. For the purpose of discussion we number the 
stages of the routers in the model as shown. All the packets in the fifo buffer of the probabilistic sink 
must be dps packets. Ifa fifo of the router in stage one contains one or more packets then the packet at 
the output of that fifo must be a dps packet. [f that fifo contains more than one packet, any packet that is 
not at the output may be a bps packet. The probability that such a packct is a bps packet is 1/2. Similar 
statements can be made about packcts buffered in higher stages. The probability that a packet at the 


output of a fifo in stage & is a bps packet is 12k VD). If that fifo contains other packets, the probability 
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lig. 10. Model Buffers Involved in Tree Buffering 
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that such a packet is a bps packet is vak : 


Thus, we can get an upper bound on £'[q,, |/,, =f; (4)] by assuming that the buffered packets are 
packed in the lower stages. We assume that at time # all of the i buffered packets are in the (f) (i))+1 
lowest stages where fyi ) is the smallest non negative integer such that the capacity of the lowest 
(f)(1))+1 stages is greater than or equal to 7. The (fpCi))+1 lowest stages include stage 0 through 


stage S5( i). Notice that 
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Xe 00 (FD pe cee S =00 fyi) Bok for 0 (20) 
and 

BZD << BQUHO)+ Dy for £>0. (21) 
As a result, 

fy(i) S$ log,(i/B +1) for i>0. (22) 


Given the discussion of the previous paragraph, the definition of f)(7), and the fact that the k th stage 
contains 2* fifos, (E'{q,, |!,, = 7 (A), the expected number of bps packets at time n given that the total 


number of packets buffered at time # is equal to 7, is less than or equal to 


B+ eto Gat (23) 
or 

(Ela, Vl, =f (ODS B+ (HMB +). (24) 
Given the relation (22), we can conclude that 

(Eldq Wy =f) (OD SB + (logy(i/B +)\(B +1). (25) 


This implies a relationship between F[q, ], the expected number of dps packets at time n, and 
EI, }, the expected value of f; (the total number of packets buffered at time n). If we substitute (25) into 
(19), we get 
Ela,1S 2% i>0 (B +(logs(i/B +B +1) PUL, =f OI. (26) 
Thus, 
Llg,1<(B + OB +)))PU, =f Ol + 2 jpyq (CB +(logy(i/7B + DB +P UL, =f, OOD- 7) 
Since logy( i/B +1) is equal to log»( i(1/B +177/)) and is thus cqual to (logy i)+ logy(1/B +1/i), 
Flan (B+ OB +I)P Ul, =f Ol 
+ Zing (B +((logy §)+logyg(1/B +17)KB +) PT, =f CO). 28) 
By (16), the definition of f;, 
Elqa,1< = i>0 (B+(B+DSA, G)+(B +I)P UL, =f, OD. (29) 


Thus, 
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Ig, 1<(B +E, |42B +1. (30) 


We assume that /'[/, } also exists in the limit as # gocs to infinity. Thus taking the limit, 


init GL Bet 
Ue Bal CP) 


Since 2/N -IN 25 IN, (15) and (31) imply that /"[7'], the expected value of the limiting distribution 


for fy (p,,). fy (the total number of packets buffered at time n), is greater than 


IN 
XOUT-IN\B +1) 2° (32) 


This can be used to obtain a lower bound on E [p’'], the expected value of the limiting distribution 
for the total number of packets buffered. Since p, > 2, - 1, 
E[p|]> ep} (33) 
where ae the limiting distribution for p,, and p,, is the total number of packets buffered at time a. 
Since exponcntiation is a convex function, (33) implies that 
eiy> 2, (34) 
Thus, from (32) we can conclude that /’[p'], the expected value of the limiting distribution for the total 


number of packets buffered, is greater than 


{ROUTINE 1? — 


This ends the discussion of the proposition. ff 


Thus, the results of the model suggest that deep tree buffering will occur in front of a slow router if 
the rate at which the router can accept packets is close to the rate at which packets that must go through 
that router are generated. If the rate at which packets are gencrated on cach of the network inputs is /N 
and if the rate at which the router can accept packets on that input is OUT then the model suggests that 
the expected value of fj (the number of packets buffered in front of that input) is greater than 


IN ne ; 
prneeereennteene tS SR ee ‘ . > the . 7 : 5 
XOUF-INXB +1) * Similarly, the model suggests that the expected number of packets buffered in 
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IN 
an % ? 
front of that inputis greater than 2 200U 7 -1N OB +1)" - 1. Inthe next section, we examine a stage of 


routers and we examine the probability that at least one of the routers of the stage has a deep tree 


buffered in front of it. 
2.2.3.3 Effect of the Last Stage 


In this section, we examine the effect that congestion in routers of the last stage of an indirect 
n-cube has on the network's operation. As we have scen in the previous section, congestion in a router of 
a given stage can easily affect many routers of an earlier stage. We now consider congestion in all the 
routers of a given stage. We use the last stage because analysis of the last stage is somewhat easier than 
analysis of other stages. There is a path from each network input to :ach router of the last stage. Thus, 


cougestion in any router of the last siage may affect all of che network inputs. 


We use a network model to study the effect of the last stage. Vhis model represents an indirect 
n-cube network with its outputs connected to perfect packet sinks. The model considers only buffering 
caused by congestion in routers of the last stage of the network. We use the modcl to study the limit that 


such buffering places on the throughput of an indirect n-cube network, 


Rather than analyze the model directly, we choose to transform the model, analyze the transformed 


model, and use the results to draw conclusions about the original model. 


Based on the results of the model, we conclude that the effect of the last stage of routers in a 


network docs not place a severe constraint on the throughput of the network. 
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Model of the Effect of the Lust Stage 


In the following paragraphs, we discuss the model: we describe the model, discuss the relationship 
between the model and the indirect n-cube network, discuss the characteristics of network buffering that 


will be studied using the model, and describe the details of the components of the model. 


‘The model is composed of nodes, probabilistic sinks, and a deterministic source. The model is 
constructed in the manner shown in Figure 11. The model for a 2-input network is constructed from a 
deterministic source and a probabilistic sink. ‘Phe model for a N-input network is constructed from a 
deterministic source, N/2 probabilistic sinks, and a tree of nodes with N/2 leaves. A tree with 2 leaves is 


one node. A tree with N leaves is constructed from two trees with N/2 leaves. 


The model reflects primarily two features of an indirect n-cube network: the probabilistic input rate 
of routers of the last stage, and the buffering capacity between cach router of the last stage and the 
network inputs. The probabilistic sinks of the model represent the routers of the last stage of the network. 
The nodes of the model represent the routers of the other stages of the network. The nodcs of the model 
are connected in a tree structure as shown in Figure 12. If the stages of the model are numbered from the 
root to the probabilistic sinks and if the total number of stages is d then cach packet in stage i of the 
model represents gd tli packets in stage i of the nctwork. Each node of the model has an input buffer 
of size B where B is the size of the buffers in the nctwork. Thus, the buffering capacity of the model 
between a probabilistic sink and the model input represents the buffering capacity of the network 


between a router of the last stage and the network inputs. 


We use the model to study the buffering of packets in an indirect n-cube network caused by routers 
of the last stage. The nodes of the model do not operate in the same manner as the routers of the 
network, Each node of the model evenly splits between its two outputs the flow of packets from its input. 


A node will instantly process a packet in its input buffer unless one of the buffers connected to its outputs 
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Fig. 11. Model of the Kffect of the Last Stage 


+ deter | deter | . 
| source | Source | Pe , 
| ae, 
} Fd ‘ 
Nv. 
its y Ny VV 


a tree with 2 leaves 


model of 


2-input / N/2 leaf 
network tree 
vn 


| prob | pre prob = 
| sink sia 


model of a N-input network 


/ 
N/2 leaf N72 leaf 
tree \ tree 


VG VV 


a tree with N leaves 


is full. ‘Thus, the buffering of packets in the model can be caused only by the probabilistic sinks and 


represents the buffering in the network caused by the routers of the last stage. 


We also use the model to examine the limit that conflict in the last stage of network routers places 


on network performance. The model accurately reflects the buffering capacity of the nctwork between a 
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last stage router and the network inputs. Thus, the maximuin input rate that can be supported in the 
model without source blocking is an indication of the limit that conflict in the last stage of nctwork 


routers places on network performance. 


The probabilistic sinks of the model are similar to the probabilistic sinks uscd in the previous 
section, Each sink contains a buffer of size B where B is the size of the buffers in the network. If at the 
beginning of a time unit the buffer is not empty then with probability .75 the sink removes one packet 
from the buffer. The average rate at which the sink can remove packets from its buffer corresponds to 
half the average rate at which a router of the last stage of the network can accept packets since each packet 
removed by the sink represents two packets of the network. It should be noted that the probabilistic sinks 


are a pessimistic model of the routers of the last stage. ‘The probabilistic sinks of the model can fail to 
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accept a packet for an arbitrarily long period of time, “Poe caters of the last stage of the network are 
capable of accepting at least one packet during cach time unit. We use this pessimistic model because it 
makes the discussion simpler and because even with such a model, our analysis suggests that congestion 


in the last stage of routers does not place a severe constraint on the throughput of the network, 


Each node of the model, in effect, evenly splits between its two outputs the flow of packets from its 
input. A node is assumed to operate instantly. Ifa node's input buffer is not empty and if both buffers 
connected to the outputs of the node are not full then the node removes one packet from its input buffer 


and places a copy of the packct in cach of the two buffers connected to its outputs. 


The deterministic source generates packets at a constant rate. The deterministic source generates 
each packet in the form of 100 subpackets. The source generates JN subpackcts in each unit of time that 
it is not blocked. JN is a parameter. The source buffers the subpackets internally until the first unit of 


time in which it can output a whole packet. Thus, the source’only outputs whole packets. 


Transformed Model 


Rather than analyze the model directly, we choose to transform the model and analyze the 
transformed model. We discuss the relationship between results for the transformed model and results 


for the original model. 
The transformations that we make are shown in Figure 13. 


The first transformation takes the bu fters that were distributed in the tree of nodes and aggregates 
them at the probabilistic sinks. The probabilistic sinks in the transformed model contain buffers of size 
(log, N)& where N/2 is the number of probabilistic sinks and # is the size of the buffers in the original 
nodes. ‘The deterministic source of the transformed model is directly connected to each of the 


probabilistic sinks. The deterministic source is blocked if any of the buffers in the probabilistic sinks are 


-4}- 


Vi ig. 13. Transformations on the Model of the Effect of the Last Stage 
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full. In cach time unit that it is not blocked, the deterministic source simultaneously generates [NV 
subpackets for each probabilistic sink. When the source has formed whole packets, it simultancously 


outputs one packet to each of the probubilistic sinks. 


‘The buffers in the transformed model have a greater independence than the buffers in the original 
model. In the transformed model, the buffering of packets caused by a particular probabilistic sink can 
affect the flow of packets into another probabilistic sink only by blocking the deterministic source. In the 
original model, the buffering of packets caused by a particular sink can affect other sinks by blocking 


nodes of the tree. 


It can be shown that the maximum input rate of the transformed model is at least as great as the 
maximum input rate of the original model. Thus, limits on the performance of the transformed model 
also apply to the original model. An upper bound on the performance of the transformed model implies 


an upper bound on the performance of the original model. 


The second transformation turns the buffer of size (logy N)@ in cach probabilistic sink into an 
infinite buffer in which we will look for occupancy of (logy N)8 packets, and the second transformation 
also turns the single deterministic source into a large number of deterministic sources with one associated 
with each probabilistic sink, We assume that each of the new deterministic sources operates in a manner 
similar to that of the original deterministic source. Each source generates JN subpackets in each unit of 
time. The source buffers subpackets until it has produced more than 100 subpackets. The source outputs 
each group of 100 subpackets as a whole packet to its associated probabilistic sink. Elowever, we assume 
that the state of cach source-sink pair is independent of the state of the other source-sink pairs. We also 


assume that the sources are never blocked. 


In the following paragraphs, we refer to the model after the first transformation as the model of the 


first transformation, and we refer to the model after both transformations as the divided model since the 


model is divided into independent source-sink pairs, 


Our analysis of the constraints on the input rate of the divided model also applies to the input rate 
of the original model. We examine the divided model to determine input rates that imply a high 
probability that at a randomly selected point in time at least one of the buffers contains at least (logy N)B 
packets. It seems reasonable to assume that such input rates can not be supported in the model of the 
first transformation since that model is similar to the divided model but has buffers of size (logy N)B. 
Thus, it seems reasonable to assume that such input rates can not be supported in the original model since 
the maximum input rate of the original model is less than the maximum input rate of the model of the 


first transformation. 


However, as we shall see our analysis of the divided model and simulation of the original model 
indicate that both models can support input rates close to the maximum rate at which a probabilistic sink 
can accept packets. Thus, our models suggest that conflict in a given stage of routers does not place a 


severe constraint on the overall throughput of the network. 


Probability of Buffering in the Divided Model 
In this section, we examine the Markov chain for the state of one of the source-sink pairs of the 
divided model and use the results to bound the probability in the divided model that at a randomly 


selected point in time the buffers in one or more of the sinks have at least (log, N)B packets. 


The Markov chain fC) for the state of onc of the source-sink pairs of the divided modct is shown 
in Figure 14. The state is equal to the number of subpackets being stored at the source plus 100 times the 
number of packets in the buffer of the sink since cach packet is composed of 100 subpackets. For the 
purpose of discussion, we use the notation Py) (tJ) to refer to the probability of a transition to state 7 
given that the chain is in state ¢. A stationary distribution, Pps: for the chain is a distribution such that 


Pps). the probability that the chain is in state /, is given by 
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Fig. 14. MCp 


WL. to UN nee -to2IN+1 


1 


q : \ Ge es 
i. 


=> to 100 + IN koe =. to OL + IN 


Pos (i) = 25> IN PHU. DPpsV). (36) 


We refer to (36) as the equilibrium equation for Png (i). 


As is shown in the following subsection, any stationary distribution, Png: for MC must be such 


that 

(ead 10+ IN VIN /15)> Ei 10010 7 Pps (i) for 42100 (37) 
and 

© = 100 jt IN-1 Pps (A) = (rad UN /75) for 2100 (38) 
where 

Ka<l 


and a is the root in this range of the equation 


N ~ 25 + 75q 100, 
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For the purpose of discussion, we define some notation. We use the notation Pub to represent the 
probability in the divided model that ata randomly selected point in time a given buffer has at least (logy 
N)B packets. We use the notation Py to represent the probability that at a randomly selected point in 


time one or more buffers have at least (logy N)B packets. 


The relations, (37) and (38), for one buffer can be used to obtain results for the overall divided 


model. 


Proposition. 


-aIN (EEE 2/8 _@ 


(39) 
where Py is the probability that at a randomly selected point in time one or more buffers have at least 


(logy N)B packets, and a is such that 
pelts (logy a) 
=) 1008 § L00(logy N)B (40) 


a 
Proof. We use (37) and (38) to show the desired relations (39), We express Piss the probability that at a 
randomly selected point in time one or more buffers have at least (log, N)# packets, in terms of P ab» the 
probability that at a randomly selected point in time a given buffer has at least (logy N)B packets. We 
bound Pub in terms of a stationary distribution for the Markov chain MC p. We then use (37) and (38) 


to get the desired bounds, (39), on Py, 


Since each connected source and sink of the model are independent of the other sources and sinks 


of the model and since there are N/2 sinks, 


Py = 1-(-P 9g), (41) 


Pap: the probability that at a randomly selected point in time a given buffer has at Icast (logy N)B 


packcts, can be expressed in terms of Png where Png is some stationary distribution for MC jp such 
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that the probability that a buffer is in state f ata randomly selected point in time is equal to Pang ( [). 
The choice of the particular stationary distribution may depend on the initial state of the buffer. Pub is 
equal to 2 i>100(logy N)B P yg () which is equal to 

1-27 =1N 1099 Pps) © j = 100 to 100(logy N)B-L Pps). (42) 
P ab is also cqual to 


(27 =100t000 Pps) © j= 100 tw 100(log, N)B-1 Pos). (43) 


Using (37), (38), (42), and (43) we can bound Pob: the probability that at a randomly selected point 
in time a given buffer has at least (logy N)B packets. Since the long term average rate of packets out of a 


buffer of the divided model must equal the rate in, 


and 
Thus, 

Poy < IN/75- (2 i 100 to 100(logy N)B-1 Pps (4). o) 
Given (38), 

Pp < UN Tisya len, NBN (47) 


(37), (43), and (44), imply that 

Pog 2 (IN /IS)(IN 175)(1-cr) 00108 N)B “LOL + IN) | (48) 
Thus, since a < 1 and JN < 101 we can conclude from (47) and (48) that 

(IN /75)a 0008) N)B-IN 99) > py > (IN /15)al lee, NDE (49) 


We introduce some notation that makes the discussion below simpler than it would otherwise be. 


In particular, we define @ to be such that 
i (logy a) 


1002 * 100(logy N)B 


( 
a=2 (50) 


It should be noted that @ is a function of /N, the source input rate, just as a is a function of JN. 
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Given (49) and (50), P ab: the probability that at a randomly selected point in time a given buffer 


has at least (logs N)# packets, is less than 
; IN +99 (IN +99\Mlog9 a) 
(log) N+" G08 + (82 4) ~~“ To0CiGgs NB 


(IN /75)2 
| < (IN/15\a/N)22/8 (51) 


and 


P op > CIN /7T5)a/N). (52) 


From (51) and (52), we deduce bounds on Pps the probability that at a randomly sclected point in 
: ; N/2, Ppp (N/2) 
time one or more buffers have at Ieast (logy N)B packets. Since 0 < Pep <1, (1-Pop) <e & , 


Thus, with a defined as above P, is greater than 
-a IN 


tech 150) (63) 
Since 0 < P eb <1, 
4A 
(Poy 7 TPN) 
je P, , (54) 


Thus, with a defined as above P,, the probability that at a randomly selected point in ume one or more 


buffers have at least (logy N)B packets, is less than 
aIN\»2/B _(IN/15)*(a/NY*(N/2)2°/ 8 
30” ) 


1-(IN /75\(a/N) 22/8 (55) 
and 
(CN g2/B g2y/B ) 
M50 Nea Br (56) 


This ends the discussion of the proposition. ll 
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Implications of the Models on Network Throughput 


The lower bound. (53), on P, for the divided model suggests a limit on the maximum input rate of 
the original model. In the divided model, P, is the probability that at a randomly selected point in time 
at least one of the buffers has at Icast (logy N)# packets. As was discussed carlicr, it seems reasonable to 
assume that input rates that imply a high Py in the divided model can not be supported in the original 
model. However, this docs not place a strong constraint on the input rate of the original model. From 


relation (53), we know that if we want Py in the divided model to be less than some value then we must 


-aIN 
choose JN such that1-¢ 159 “is tess than that value. As N goes to infinity, if@ goes to zero and IN > 
-aIN -a IN) 
Lthen1]-¢ 150 goes to zero. As N goes to infinity, ifa gocs to infinity and /V > | then l-e 150 


gocs to one. Thus to find an upper bound on /N in the divided model for a particular P,, in the limit as 
N goes to infinity, we assume that @ approaches a constant independent of N. In such a case, equation 
(S0) implies that a approaches 21711008) the corresponding value of JN can be deduced from the 
equation, alN = 25 4+ 5a '90, For example, if B is equal to five then a must be less than 2° 17500. 
21/500 is approximately equal to .998615. This implies an upper bound on /N of 73. For comparison, 
the upper bound on /N implied for B equal to one is 67, the bound for B equal to two is 71, and the 
bound for B equal to ten is 74. Obviously, these upper bounds are close to the upper bound of 75 placed 


by the fact that, as was discussed in the description of the original model, a probabilistic sink of this 


section can only accept packets at an average rate of .75 packets per unit time. 


In fact, it secms that if the buffers of the original model have at least modest size then the average 
input rate of the model can be close to the average rate at which packets can be removed from the buffer 
of a probabilistic sink. We have simulated the original model for several buffer sizes. The model was 
simulated with both $12 probabilistic sinks and with 1024 probabilistic sinks. A simulation run was made 
for each combination of model width and buffer size. In cach run, the deterministic source was capable 


of generating a packet in cach time unit that the source was not blocked. Thus, the rate at which packets 
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were generated by the source was determined by blocking. The buffers in the model were full at the 
beginning of cach run. For each run, we measured the number of packets that were generated by the 
deterministic source during the run. The results are shown in ‘Table I. For each combination of model 
width and buffer size, the average rate at which packets were generated is listed. As can be seen from the 
table, the results for 512 sinks are close to the results for 1024 sinks. For both scts of results, the average 


input rates are not far from .75 packets per unit time for even modest buffer sizes. 


Thus, our study of the original model suggests that slow routers of the last stage in an indirect 


n-cube network do not place a severe constraint on the throughput of the network. 
The Stationary Distribution for the Divided Model 


We now return to the detailed analysis of the divided model in order to show the bounds, (37) and 


(38), on any stationary distribution for the divided model. 


Since direct analysis of MCp, the Markov chain for the divided model, scems difficult, we 
indirectly analyze it by comparing it to two simpler chains. The first of these chains climinates the first 
100 states of the original chain since we are primarily interested in the later states of the original chain. 
The second chain is easier to analyze than the first chain and gives information about the first chain and 


the original chain. We introduce each of these chains and show the the relation between the stationary 


Table I. Simulation of the Model of the Effect of the Last Stage 


B 1 2 3 5 10 
av. in for 512 Probabilistic Sinks 50.8 64.0 67.6 70.6 72.6 


av. in for 1024 Probabilistic Sinks 50.7 63.5 67.6 70.6 72.8 
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distributions of each and the stationary distributions of MC py, the Markov chain for the divided model. 
We obtain the stationary distribution for the second chain and use it to bound the stationary distribution 


for MC p. 


The Weeiicay chain MC py shown in Figure 15 is very similar to the chain for the divided model 
with the exception that the states less than 100 have been removed. For 100 < i < 199-/N, wrap(/) is 
equal to the smallest j such that for some integer k, KIN + i -100 + IN = j and j > 100. wrap(/) 
is the first state greater than or equal to 100 that would be reached after a transition down from state 7. 
We use the notation pp, (i,j) to refer to the probability of a transition to state j given that the chain is in 
state i. A stationary distribution, Php 1S? for the chain is a distribution such that Pos) the 
probability that the chain is in state /, is given by the equation 

Ppis©) = 25> 100 ep OP pisV). (37) 
We refer to (57) as the equilibrium equation for Pp ;g (i). The equilibrium equations of MC; , are 


Fig. 15. MCpy 
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very similar to the equilibrium equations of the divided model. 


This similarity can be scen more clearly if Pry UN) through P yyy (99), the stationary probabilities 
for states less than 100 in the divided model, are climinated from the equilibrium equations of the divided 
model. We can eliminate Png UN) by using the equation for Pryg UN) to substitute for Pyg CIN) in 
the remaining equations. ‘This process can be continued for Pog Un +1) through Pryg (99). The 
resulting equation for Ppyg(i) for cach ¢ > 100 corresponds directly to the equilibrium equation for 
Pp 754). Ta particular, if we take the equation for Pyyg (i) and map Pyyg (j) to Py y (/) for cach j > 
100, we obtain an equation which is identical to the equilibrium equation for Pps). From this we 
can conclude that for any stationary distribution Png for the Markov chain MC, the chain for the 
divided model, there exists a stationary distribution P DIS for the Markov chain ALC Di such that for 


fay 


SOC COUStaut C aia for ali i > 100, Png = P yg (4). Since Z i>100 Pps) = i, we can 


l 
conclude that c = => and that 
~i>100 Pps) 
F 1 : 
Pye (tl) = ao ms PP pyc). (58) 
DIS = §>100 Pps) DS 


The chain A/C) shown in Figure 16 is related to MC p, and is therefore also related to MC jp, 
the chain for the divided model. We examine MC)» because it is related to MC p and because, as we 
shall see later, the stationary distribution for MC p) is casy to determine since there is a simple geometric 
relationship among the probabilitics of the various states. We use the notation pj) (i,/) to refer to the 


probability of a transition to state / given that the chain is in state 7. If @ is such that alN = 95 + 


7500 then 
Pp lid) = F5 al) 100) for 100< 1 <199-IN and 100<j<99+IN, 
ppp iit IN) = 25 for 1>100, 
Py (i, i-1004+ IN) = .75 for i >200-/N, 

and 


Ppj(ij) = 0 otherwise 
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Fig. 16. MCpp 
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The stationary distribution Pnos for the Markov chain MCp) is the distribution such that Ppog (1), 

the probability that the chain is in state 7, is given by the equation 
Poas() = 27> 100 Ppa. OPpas()- (60) 

The transitions of the chain MCp) are similar to those of the chain MCp, . Many of the possible 

transitions of MfCpy have the same probabilities as the corresponding transitions of MCp,. In 


particular, 
Pp? (i.it+IN) = pp, (i,i+IN) = .25 for (>100 


and 
Ppgi,t- 100+ IN) = Pp (i,t-l00+ IN) = 75 for §>200-IN . 
(61) 


The transitions of MC), from i to j where 100 < i < 199-N and 100 < j < 99+/N differ from the 
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corresponding transitions of MC py, . But, forany / such that 100 < 1 < 199-/N, 


= =100 0994 IN PDZUS) = = =100w 994 IN PDs (iS) = O75). (62) 


We show that any stationary distribution Pris for the chain MCh) is related to the stationary 


distribution P nos for the chain MC pp. 


Proposition. 
2; =100t0 ;+IN-1 Ppas) 2 ~;=100t0 7; Pais) (63) 
and 


2;=100to j+/N-1 Pos) 2 ~;=100t0 7 Ppas):- (64) 


Proof. This can be scen by comparing the operation of the two chains during a long period of time. For 
the purpose of discussion, we define Pp (t, @) to be the probability that chain MC, is in state i at 
time t. We define Prot, i} to be the probability that chain MC p> is in state 7 at time ¢. We assume 
that for all />100, Pp} (1, é) is equal to Pp, i) and is also equal to Porg() where Pris is the 


given stationary distribution for the chain Af{Cp7. 


We show a relationship between Py, and Pp that exists for all ¢. Using this, we show a similar 
relationship between Pp; g, the given stationary distribution for the chain MCp;, and Ppog, the 


stationary distribution for the chain MC p>- 


In particular, we show by induction on ¢ that for all £90 and j > 100 


=; =100to f+IN-1 p22 2; =100t0 7 Pos O (65) 


and 


®;=100t0 f+ IN-1 Poi 92 2 j= 10010 7 Ppa 9- (60) 
Clearly, these relations hold for ¢=1. We show the case for £>1 by considering the transitions of each 


chain. From the transitions of the chain AfCp 7, it can be shown that for ¢>1 and j > 100 that 
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*i=1000 jf Poss 9) 
S-Di = 100 199-1N Po Old) 
+ 252; =100t0 j-IN Patel O 
+ 15% j =200-1N wo j+to0-IN Ppl) oD 
or 
=; = 100 tof Pry (t, i) 
S12 j= 1000 f+1001N Poth OD 
+ 25% 3100 0 j-n Pps (Cb a), (63) 
and that 
*j=100t0 f+IN-1 Pps) | 
= 715% j= 100 to 199-IN Pps 2) 
+ 252 i= 10000 j-l Ppl. O 
+ .75E j=200-IN to j +99 Ppl D iS?) 
or 
=; =100to j+IN-1 Pps 
= 152 i= 10000 {+99 Poth dD 
+ 25% 5-100 10 j-1 Ppy(eh (70) 
Similarly, from the transitions of the chain MC’, it can be shown for £>0 and j > 100 that 
= j=100t0j Por) 
S-% j= 1000 j+100-1N Pp2-l 
+ 25% 1900 j-IN Pog), 
and that 
*j=100t0 j+IN-1 Pog.) 
= -D® i= 1000 7 +99 Pp) O 
+ 25Ej 100 to jl Pyke hd. (72) 


Thus, given the hypothesis of induction we can conclude that the desired relations, (65) and (66), hold for 
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>1. Asa result, we can conclude that the desired re’ ticis hold for all 4>0. In other words, for 4>0 and 


J = 100 
~;=100t0 ;tIN-1 Pp. 9 2 Zp=100w j Pps. O (73) 
and 


2; =100t0 ft IN-1 Poi O 2 Z=1000 j Pos 9. (74) 


Since Py, (f, 1) equals Pog (O for (>0 where Pris is the stationary distribution chosen 
above for the chain MC p, and since in the limit as ¢ goes to infinity P p(t, £) gocs to Pros) where 


Pros is the stationary distribution for the chain MC yp), we can conclude that 


2; =100t0 ;+IN-1 Pp2s) 2 %;=10000 7 Pais (75) 
and 
~j=100t0 j+IN-1 PoIs() 2 %j=100t0 5 Po2s')- (7) 


This ends the discussion of the proposition. ¥ 


We can relate any stationary distribution Pros for MCp, the chain for the divided model, to the 


stationary distribution Prog for the chain MC pp. 


Proposition. 

*;=100t0 7 +IN-1 ON/PPp2g() 2 Xi -r0nt0 7 Post) — for j2100 7) 
and 

%j=100t0 f+IN-1 Pps) 2 2; = 100007 UN/Ppzg(4) for f2 100. C78) 


Proof, From (58), we know that for any stationary distribution Png for MC p there exists a stationary 


distribution Pjy7.¢ for the chain MC py; such that for i> 100 


a partes ERC (79) 
Dis 2 F>100 Pos) PS 


Since the long term average rate of packets out of a buffer of the divided model must equal the rate in, 
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Pps) = UN7TS)P py 6) for (>100. (80) 
Given the relationship between any stationary distribution for M/C jy, and the stationary distribution for 


MC yy). we can conclude that 


2; =100t0 s+ IN-1 UNDP p29) 2 Zi =100t0 5 Post) — for j2100 (81) 
and 
2; =100t0 j+IN-1 Pps() 2 Fj =100t07 ON p2g() for fz 100. 2) 


‘This ends the discussion of the proposition. # 


‘The stationary distribution for the chain MC, can be easily obtained. The equilibrium equations 


for MC pr are 


l-a : 
IN j = 100 to 199-1N P28) 


fur 100 i $59+ IN 


Pps (i) = TSP 95 (i +100-IN) + Tah TOO 
and 

Prpag Ci) = TSP p95 (i + 100-IN) + 25P pag (i- IN) for i >100+1N 

(83) 

where 

Kad 
and a is the root in this range of the equation 

a!N = 25 + 75q10, 
From the equilibrium equations, it can be shown that 

Ppog(i) = a! Mp, (100) for £100. (84) 
Since X i> 100 Prog ( i) = 1, we can conclude that 

P pg (100) = (l-a). (85) 
Thus, 


Pog (i) = a! 0) for i> 100. (86) 


- 5Y- 


From (86), the solution for the stationary distribution Pros for the chain MC py), and (81) and 


(82), the relations between Prog and any stationary distribudon Png for the chain of the divided 


model, we can conclude that ; 

(eal FIN YIN 75) > Ej 1000 j Pps) for j= 100 (87) 
and 

= j= 100 of tIN-1 PDs (1) = (rad PIN 75) for 7 > 100 (88) 
where 

Kad] 
and 


a! = 25 + 75q100, 
2.2.3.4 Interaction of Stages 


In the following paragraphs, we examine the intcraction of routers of different stages. In the 
previous paragraphs, we examined the effect of conflict at one router when conflict at all other routers 
was ignored, and we examined the effect of conflict in a given stage of routers when conflict in all other 
stages was ignored. Now we consider the effect of the intcraction of routers in various stages of the 


network. 


The discussion has two parts. 


In the first part, we consider all the routers along a given path through the network. The diffusion 
of packet flow that occurs along such a path duc to the interaction of routers of the path seems to be one 
of the primary factors constraining the overall network throughput. We cxamine the interaction of 
routers along a path by ignoring conflict at routers that are not on the path. As we shall sce, the 
interaction of routers along an infinitely long path still allows a nonzero flow into the path. Since this 


type of interaction between stages represents the strongest constraint on overall network throughput that 
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we find, our study of such interaction suggests that the normalized throughput of the indirect n-cube 


network approaches a nonzero asymptote as the size of the network goes to infinity. 


In the second part of the discussion, we consider the interaction of routers in a tree of the network. 
The interaction among such routers can be quite complex. We examine once particular type of interaction. 
While this interaction docs not seem to have an important effect on the overall throughput of the 
network, it may cause a few of the routers connected to the network inputs to be slow for a long period of 


time. 
Interaction of Routers Along a Network Path 


We study the flow of packets along a typical path of the indirect n-cube network using the model 
shown in Figure 17, The model represents a path through the network. The model allows us to study the 
effect of conflict at routers along the path in the absence of conflict at other routers. We first describe the 


model, then we analyze it using simulation. 
Model of a Network Path 


The model reflects the interaction of the routers along a path of the indirect n-cube network. The 
modcl ignores the interaction in the network between a routcr on the path and any router not on the path. 
The model contains a sequence of 2-input 2-output nodes. The nodes of the model represent the routers 
along the network path. As shown in Figure 17, the first output of each node of the model, except the last 
node, is connected to the first input of the next node. The second output of cach node is connected to a 
perfect sink. The second input of cach node is connected to a probabilistic source. The first input of the 
first node is connected to a probabilistic source. The first output of the last node is connected to a perfect 
sink. The connections of the second input and second output of each node represent the connections of 


the corresponding router of the network to routers not on the path. The probabilistic source connected to 
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Fig. 17. Model of a Network Path 
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the second input of the node provides a steady flow of packets into the node. The rate of flow can be 
adjusted to equal the average flow into the corresponding input of the corresponding router. The perfect 


sink connected to the secend output can not block. Thus, congestion in the model can be caused only by 
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conflict in the nodes and represents the congestion along the network path due to conflict in the routers 


of the path. 


Each node of the model has a buffer on cach of its inputs. ‘The buffer on the first input has size B. 
The size of the buffer on the second input is several times #. The long buffer of the second input ensures 


that short term variations of the node do not affect the probabilistic source connected to the input. 


The operation of each node of the model is similar to the operation of a router in the indirect 
n-cube network. Each packet entering the node is assigned a one bit tag which determincs its route 
through the node. The tag is randomly selected with zero and one being equally likely. If the tag is zero, 
the packet must leave on the first output of the node. If the tag is one, the packet must Icave on the 
second output of the node. In cach unit of time, the node attempts to transfer a packet from cach of its 
input buffers. For each input buffer, the node attempts to transfer the first packet, the packet that 
entered the buffer first, to the output that corresponds to its tag. The packet is transferred if the buffer 
connected to the desired output is not full, and if there is no conflict from the other input buffer or if the 
packet wins the arbitration of the conflict. In the case of conflict, the node randomly selects a packet to 
transfer. The two possible choices are equally likely. The node will transfer at most one packet from each 


of the input buffers in a unit of time. 


The probabilistic sources and perfect sinks used in this model are similar to the corresponding 
devices used in the previous paragraphs. ‘The probabilistic sources produce packets. If the input buffer 
connected to a probabilistic source is not full at the beginning of a time unit then with some probability 
the probabilistic source places an additional packet in the buffer. The probabilistic source connected to 
the first input of the first node generates packets with probability JV. ‘The probabilistic sources 
connected to the sccond inputs of the nodes generate packets with probability S/. The perfect sinks 


never block and accept packets at whatever rate they are presented. 


Simulation of the Model 


We use simulation of the model to study the effect of conflict in a randomly selected network path 
on the rate at which packets can enter the first router of the path. In this section, we describe the 
simulation of the model, discuss the unplications of the simulation results, and compare the simulation 


results for the model to simulation results for the complete indirect n-cube network. 


We run the simulation in such a way that we can use the results of the simulation to draw 
conclusions about the limit that conflict in a randomly selected network path places on the rate at which 
packets can enter the first router of the path. In the model, we sect JN to 1. We examine the rate at which 
packets enter the first input of the first node as a function of the value of S7. We find a value such that 
when S/J is sct to the value, the rate at which packets enter the first input of the first nodc is also equal to 
the value. We refer to this value as the maximum input rate for the model. The maximum input rate for 
the model in some sense represents the limit that conflict in a randomly selected network path places on 
the rate at which packets can enter the first router of the path, if conflict elsewhere in the network is 


ignored and if it is assumed that each network input receives the same input rate. 


We have simulated the model for several values of 8 and for several path lengths. The maximum 
input rate for cach case is shown in Table II. It should be noted that the valucs listed are percentages and 


that only values with whole percentage points were used in the simulation. 


The model suggests that the effect of conflict in a randomly selected path increases with the length 
of the path. The maximum input rate for the model decreases as the length of the model increases. Each 
node can block all carlier nodes. Nodes at the beginning of a long path can be blocked by any of the later 
nodes. Thus, the maximum input rate for the model of a long path is less than the maximum input rate 


for the model of a short path. 
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Table If. Simulation of the Model of a Network Path 


B=5 length maximum input rate 


128 59 
64 60 
32 61 
16 63 
8 64 
4 67 
2 71 


64 aL 
32 54 
16 56 
8 59 


However, the model suggests that for very long paths the effect of conflict increases very slowly 
with path length. The chance in the mcdel of a long path that all of the buffers between a late node and 
an early node are full is small. The chance that a conflict in the late node blocks the early node is small. 
While we will not discuss this issue in detail, we expect that the maximum input rate for the model 
approaches a nonzero asymptote as the length of the model goes to infinity, As is shown in Table II, 
128-node models were simulated. We expect that the maximum input rates for the 128-node models are 


close to the maximum input rates for infinite length models. 


The limit of the input rate of a randomly selected network path implies a limit on the overall 
throughput of a network. Clearly, we do not expect the total throughput of a network to be greater than 
the width of the network times the maximum input rate of a randomly sclected network path. For most 
networks, this limit is stronger than any of the other limits studied so far. For example, this limit is 
usually stronger than the limit placed by the slowest router in the last stage when conflict in other stages is 


ignored. However, it is important to note that this limit does not rule out high throughput for very large 
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networks. “The ratio of this Timit to the number of network inputs approaches a nonzero asymptote as the 


size of the network goes to infinity. 


We have not found any factors that place a significantly stronger constraint on network throughput 
than the interaction of routers along network paths discussed in the previous paragraphs. This suggests 
that the normalized throughput, throughput divided by the number of network inputs, of the indirect 


n-cube network approaches a nonzero asymptote as the size of the network goes to infinity. 


For comparison, complete indirect n-cube networks were simulated. The results are shown in Table 
III. While the simulation results do not clearly indicate the normalized throughput of networks of infinite 
size, the normalized throughputs of the networks simulated are consistent with the results of the model 
above since the normalized throughput of each complete network is less but not drastically less than the 


normalized throughput of the corresponding model. 
Interaction of Routers in a Tree 


In the following paragraphs, we consider the interaction of routers in a tree of routers in an indirect 
n-cube network. The interaction among such routers can be quite complex. We examine one particular 


type of interaction and its effect on the behavior of the network. While this intcraction does not seem to 


Table HI. Complete Network Simulation 


B=5 
d 1 2 3 4 s) 6 at 8 9 10 ll 
N 2 4 8 16 32 128 64 256 512 1024 =. 2048 


normalized .749 681 = 643617 598 S837 562 553 548 542 
throughput 
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have an important effect on the overall throughput of the network, this interaction may cause a few of the 


routers connected to the network inputs to be slow for a long period of time. 


The effect of this interaction is important in networks of modest size, networks with Iess than 10 
stages. AS was mentioned above, this interaction may cause a few of the network inputs to be slow for a 
long period of time. We develop a model for this interaction, use the model to estimate its effect, and 
check our estimate by simulating the whole network. Our model suggests that if this type of interaction 
occurred in arbitrarily large trees, its effect would asymptotically grow as the square of the depth of the 
network. In fact, this interaction only occurs in trees of modest size, but it is strong enough to cause the 
input rate for the slowest input of a network with cight or nine stages to be less than half the expected 


input rate for a randomly sclected input for a period of forty units of time. 


The type of interaction discussed in the previous paragraph is less important in very large networks, 
networks whose depths are much greater than 10 stages. Since that type of interaction docs not occur in 
very large trees, other factors become more important for very large networks. We bricfly consider one of 
these factors. For very large networks, as we shall see, this factor implics that the slowest input router 
requires greater than cgn/(logs n) time to accept 3B packets where cg is aconstant, B is the buffer size, 


and n is the depth of the network. 
Trees in Networks of Modest Size 


In a network of modest size, we examine a particular intcraction of routers in a d-stage tree whose 
leaves are connected to the network inputs. We select a router of the n-d th stage from the final stage of 
the network where n is the depth of the network. We refer to this router as the root router of the tree. 
We consider the tree composed of the root router and all of the routers that can direct packets to that 
router as shown in Figure 18. We refer to the routers of the tree in the d-1 st stage from the root router as 


the leaf routers of the tree. Below, we study the time required by the slowest leaf router to accept cy RB? 
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Fig. 18. d-Stage Tree in a n-Stage Network 


Voy: 


packets where # is the size of the input buffers of the routers and Cy is aconstant. We do this using a 
model of the tree. We introduce the model and analyze it in order to estimate the time required for the 
slowest leaf router of the model to accept cB? packets. We use simulation to compare our estimate to 


the performance of the model and to compare our estimate to the performance of the d-stage tree. 


We assume that conflict in other parts of the network affects the tree only at the root router. It 
seems likely that the leaf routers accept packets more quickly with this assumption than without it. Thus, 
we make this assumption in order to estimate an upper bound on the rate at which the slowest leaf router 


accepts packcts. 
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We assume that the links connected to the outputs of the root router accept packets very slowly. 
We assume that the inputs of the network are connected to perfect sources that supply packets as quickly 
as they can be accepted. We assume that n, the depth of the network, is no more than 10. We assume 
that the rate at which packets can be accepted from the outputs of the root router is limited by conflict in 


the later stages of the network and is less than the rate at which the root router can supply packets. 


We are interested in the behavior of the tree after the network has been in operation for some 
period of time. We randomly choose a point in time after the network has been in operation for a long 
time. We examine the duration of the shortest period beyond that point such that during the period each 
leaf router of the d-stage tree of the network accepts cB? packets where 8 is the size of the input 


buffers of the routers and Cy is aconstant. 
Model of a d -stage Tree 


In order to estimate the duration of this period, we study the model of a d-stage tree shown in 
Figure 19. The characteristics of the model correspond for the most part to the assumptions shat we made 
above. The characteristics of the model are such that it seems reasonable to assume that the model will 
Icad to a lower bound on the duration of the period. We refer to the model as the d-stage special tree or 


simply the d-stage special. 


The d-stage special is composed of routers, probabilistic sinks, perfect sinks, and perfect sources as 
shown in Figure 19. The routers operate in a manner similar to that of the routers of the network. Both 
of the outputs of the root router are connected to probabilistic sinks. Each probabilistic sink has an input 
buffer of size B. If the input buffer of a probabilistic sink is not empty at the beginning of a time unit 
then with probability cy the sink removes a packet from the buffer where c9 is a small constant. Each 
router of the tree except the root has one output connected to another router of the tree and one output 


connected to a perfect sink. We refer to the routers of the tree in the d-1 st stage from the root router as 
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Vig. 19. d-Stage Special 
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the leaf routers of the tree. The inputs of the leaf routers are connected to perfect sources that produce 


packets as quickly as they can be accepted. 


Analysis of the Model 


We examine the behavior of the d-stage special after it has been in operation for some period of 


time. We randomly choose a point in time after the d-stage special has been in operation for a long 


period. We examine the behavior of the d-stage special after that point. 
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We are interested in the time required after the selected point for the slowest leaf router to accept a 
total of C] Be packets. for the purpose of discussion, we refer to this time as the slowest leaf router 
acceptance time, Ty. As before, we use the notation f'[x ] to refer to the expected value of x. Below, we 
estimate a lower bound on E[T 4). We refer to this estimate as estT ,. We develop a recursive definition 
for es(T 1. The definition is given in equations 99-101. For large d, as we shall see, es/T 7 grows as the 
square of d. We arc primarily interested in estT 4 for d < 10 since we expect the type of interaction 
modeled by the d-stage special to occur only in trees of depth < 10. As is shown below in Table IV, 


estT 1 grows rapidly even for d in this range. 


In order to make the desired estimate of ET gh the expected value of the slowest leaf router 
acceptance time, we consider the behavior of one router from cach stage. In each stage, we select the 
fouler that accepts Wie smallest fuuiber of parkeis in the 7 unit period during witich the siowesi icaf 
router accepts cB? packets. For 0 < i < d, we refer to the selected router of the / th stage from the 
root router as the é th intermediate. For 0 < i < d, we define numbpackets 4 to be the number of 
packets accepted by the / th intermediate in the T , period. In order to estimate L’ [numbpackets 9, the 
expected value of numbpackets 9, we consider £ [numbpackets a) -E [numbpackets || for each value of 
i such that 0< 7 < d and we make use of the fact that E[numbpackets 4} is equal by definition to c, B 2. 
We use E{numbpackets 4), the expected number of packets accepted during the period by the root 
router, to estimate [T ak the expected length of the period. We arguc that E[numbpackets 9] is big and 


then assuming that it is big we argue that [7] is big. 


For cach value of i such that 0 < i < d, we estimate F [numbpackets 7) - E[numbpackets 4 the 
difference between the expected number of packets accepted by the /-1 st intermediate and the expected 
number of packets acccpied by the / th intermediate, by considering the /-1] st intermediate and the two 
routers connected to its inputs as shown in Figure 20. For the purpose of discussion, we refer to the 


routers connected to the inputs of the /-1 st intermediate as the input routers. We consider the operation 
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hig. 20. ¢-T st Intermediate and input Routers 
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of these components in the T 4 unit period after the selected point in time. 


The packet tags examined by each of the input routers correspond to a Bernoulli process and the 
packet tags examined by one of the input routers are independent of the packet tags examined by the 
other input router, and we use these facts in the following paragraphs to estimate a lower bound on 
PE [numbpackets oy - E[numbpackets AL We argue that during the period one input router is likely to 
receive a larger fraction of packets labeled for the i-1 st intermediate than the other input router receives. 
We then argue that during the period one of the input routers ts likely to accept a smaller total number of 
packets than the other input router accepts. Since the number of packets accepted during the period by 
the ¢ th intermediate can be no more than the number accepted by the slower input router of the i-1 st 
intermediate, we estimate a lower bound on F [mwnbpackets ra - E[aumbpackets 4 by estimating the 
difference between / [numbpackets oT and the expected number of packets accepted during the period 


by the slower /nput router of the 7-1 st intermediate. 


For the purpose of discussion, we define some notation. We arbitrarily order the iupur routers. 
Fork = Land & = 2, we refer to the following quantities, 
PA,. the number of packets accepted during the period by the k th inpur router, 
TPA ,, the number of packets accepted during the period by the & th input router that are tagged 
for the i-1 st intermediate, 
TPO x: the number of packets output during the period by the & th input router that are tagged for 
the 7-1 st intermediate, 
MPA, the minimum of PA l and PA > 
TPA’,, the number of packets that are in the first APA packets accepted by the k th input router 
and that are tagged for the /-1 st intermediate, 
Le TPA, /PA,, 
and 
DBs TPA’, / MPA. 
We also define kmaxtpa to be equal to one if TPA 71 is greater than or equal to TPA 4 and we define 


kmaxipa to be equal to two otherwise. 


About half of the packets accepted during the period by an input router are tagged for the i-1 st 
intermediate. The tag of cach packet is independent of the tags on other packets. ‘Thus, the tags on the 
packets accepted by the router can be considered to correspond to a Bernoulli process. The chance that a 
packet accepted by the router is tagged for the /-1 st intermediate is (.5). Thus fork = Land k = 2, 
L{[TPA’ x }, the expected number of packets that are in the first 4f/PA packets accepted by the & th input 


router and that are tagged for the i-1 st intermediate, is equal to (1/2) [A/PA ]. 


However, since there are two input routers and since the tags on packets received by the two input 
routers correspond to two independent Bernoulli processes, 


EATPA’ pra xtpal? Q/2)E [MPA] + 6(EIMPA pl/2 (89) 
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for some constant C3. 


Thus, [TPA kmaxtpah the expected number of packets accepted during the period by the 


Amaxtpa th input router that are tagged for the /-1 st intermediate, is greater than 
(1/2)F [MPA] + cx(E [MPA p72 + (1/2)E [PA kmaxtpa ” MPA}. (90) 
Assuming small deviations from the means, we assume that ETS kmaxtpah the expected value of 


TP /PA kmaxtpa’ is greater than 
eg(E [APA D2 
te 
E|PA kmaxtpa| 

for some constant cg. We assume that [PA kmaxtpa| -E[MPA]K E[PA kmaxtpa| and that as a result 


A kmaxtpa 


(91) 


c 
EUS tmaxtpa! DLL ia 3 [72 for some constant cs. ‘Thus, we assume that 
(E| kmaxtpa ) 
41/9 Beers Ae ee (92) 


EV, }>1/72 
ww Kmaxtpa yi72 


ws 
(ETP A kmaxtpal 
for some constant CG: 


To simplify the discussion, we assume that the input buffers of the 7-1 st intermediate remain non 
empty during the period. The motivation for this assumption can be scen by examining the operation of 
the tree during the period. We assumed earlier that the links connected to the outputs of the root router 
of the tree accept packets very slowly. Thus, we assume that the root router accepts packets very slowly. 
We expect the i-1 st intermediate to accept packets no more quickly than the root router since the /-1 st 
intermediate is the slowest router of the 7-1 st stage from the root. ‘Thus, we assume that the inpué 
routers can supply packets quickly cnough that the input buffers of the 7-1 st intermediate remain non 


empty. 


Since we have assumed that the input buffers of the /-1 st intermediate remain non empty during 
the period, the number of packets removed from cach of the input buffers of the 7-1 st intermediate 
during the period is independent of the tag bits examined by the input routers and is thus independent of 


kmaxtpa, and these facts can be used to bound [TPO kmaxtpa ]. the expected number of packets 
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output during the period by the Amaxipa th input router that are tagged for the f-1 stintermediate. The 
expected number of packets removed during the period from the Amaxtpa th input buffer of the f-1 st 
intermediate is (1/2)F [numbpackets 44, However, the number of packets in the Amaxtpa th input 
buffer of the /-f st intermediate at the end of the period may depend on the tag bits examined by the 
Amaxtpa th input router during the period. Thus, the following relation holds for /[7PO kmaxtpa |, 


(1/2) [aumbpackets | Y-B < EUT 1S (1/2E[numbpackets FY+B. (93) 


PO kinaxtpa 


Based on the arguments of the previous paragraphs, we estimate a lower bound on 
f:{numbpackets : F[numbpackets 4], the difference between the expected number of packets 
accepted during the period by the /-1 st intermediate and the expected number of packets accepted 
during the period by the / th intermediate. Clearly, E[TPA kmaxtpa ], the expected number of packets 
accepicd duriig tie period by the Amuaipa Ui inpud tourer and tagged for dhe ¢-1 se iutenmediace, is tess 
than or equal to ELTPO gnaxipa! + 2B. From (93) and (92), we have F {TPO kmaxtpa! < 

z0 


(E[TPA kmaxtpa 


(1/2) [numbpackets +B and EU pnaxipal > V2 + Assuming small 


pl/2 , 


devtions tf se “Tp a E[TPA kmaxtpa! = 
eviations from the means, we assume that E'[PA paying] = E . Thus, we conclude 
P otf kmaxtpa!| 


that 
E{TPA a 
E|PA kmaxtpa\< ae . ’ (94) 

EITPA Me 

(ET kmaxtpa ) 

(1/2) [numbpackets eal +38 

ed ee 28 oe ee 
- r=] 172 

((1/2)E [numbpackets y ]+38) 


ELPA kmaxtpal § (95) 


eg fh [numbpackets i I 


J ((1/2)4 [numbpackets ve N4 3B yi/2 
kK E'[numbpackets 4 ]+ ee eo (96) 
J 


ae =| 172 
((1/2)h [numbpackets "]4+3B) 
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E[PA kimaxtpa 


and 
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L[PA kmaxtpa ]< E[munbpackets Teal Ee [numnbpackets ye (97) 
for some positive constant ¢7. Clearly, E’[numbpackets jy) the expected number of packets accepted 
during the period by the / th intermediate, is no greater than the expected number of packets accepted 
during the period by the slower input router (the slower of the two routers connected to the inputs of the 
i-l st intermediate) and thus is no greater than E[PA kmaxtpa |. ‘Uhus, F [awnbpackets es - 
E-[numbpackets Ay > cE [numbpackets by Yy'/. Since '{numbpackets fy} >E [numbpackets 


E [numbpackets a - E-{numbpackets 4 > ¢7(F [numbpackets Pas a (98) 


Based on the arguincnts of the previous paragraphs, we estimate a lower bound on 
E [numbpackets 4. the expected number of packets accepted during the period by the 7 th intermediate. 
We usc the notation estapkts y to refer to the estimate for a lower bound on E[numbpackets i) We 
define estnpkts A recursively. The basis comes from the fact that, by the definition of the period, 
E[numbpackets 4) is equal to cB 2 ‘The recursive step comes from the discussion of the previous 
paragraphs. Thus, we define 

estnpkts 4! = éB* (99) 

and for0< i<d, 


1 


estnpkts Ve = estnpkis | + colesinpkts by”? , (100) 


For very large values of d, esinpkts 9 grows roughly as the square of d; we are primarily interested 
in values of d less than 10, but estupkis also grows rapidly for d in this range. Below, we assume 
example valucs for cj, cz, and B, and compute estnpkis) for values of d less than 10. The results are 


listed in Table IV. 


We use esinpkts 4 to estimate a lower bound on E[T 4], the expected value of the slowest leaf 
router acceptance time. Since the zeroth intermediate, the root router, is expected to accept 


E'[nunbpackets 9] packets during the period, we assume that it is expected to output greater than 
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(1/2) :{munbpackets 9) + eg(E [numbpackets 9/2) packets on one of its outputs during the period for 
some constant ¢g. We assume that it takes (172) 17e9) E {numbpackets 4 + eg [aumbpackets 9 ))/?) 
time for the probubilistic sink connected to that output to accept the packets where Cy is the parameter of 
the probabilistic sink. We use the notation eT to refer to the estimate for a lower bound on E[T gl. 


We define 


est 4 = (1/2\(1/ey\(estnpkis®, + cglesmpkts9)'/), (101) 


As we shall see in the example below, est 4, our estimated lower bound on the time requircd for 
the slowest leaf router to accept a total of c) B 2 packets, grows rapidly with d. As we shall see, estT 4 for 


d equal to nine may be several times the size of estT , for d equal to one. 
d q 


Pualuation af the Model 


We use simulation to evaluate how well the behavior of a special tree corresponds to our estimates, 
and how well the behavior of a tree in an indirect n-cube network of modest size corresponds to the 
behavior of a special tree. As is discussed below, our simulations suggest that the time required by the 
slowest leaf router of a d-stage special to accept cB? packets grows at least as fast as estT 4 defined 
above (101), and our simulations suggest that the slow leaf routers of a tree in an indirect n-cube network 


of modest size are at Icast as slow as the slow leaf routers of a special tree of corresponding size. 


We simulated special trees of various depths and examined the slowest leaf router of cach tree in 
order to compare its behavior to our estimates. In the simulation, the size of the buffers, B, was equal to 
five. ¢>, the parameter of the probabilistic sinks, was set to (35). A simulation run was made for cach 
tree depth. Each simulation run was divided into many periods. Each leaf router accepted more than 25 
packets during cach period. For cach period of each simulation run, the me required by the slowest leaf 
router to accept a total of 25 packets was measured. The average over all the periods of a simulation run 


was computed. ‘The results of the simulation runs are shown in ‘Table [{V. For cach simulation run, the 
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Table TV. Evaluation of estT 4 


d } 2 3 4 5 6 7 8 9 10 
T 4 from 36.35 458 Stl 611 79.5 885 94.1 112.2 

simulation 

estT 4 41.1 47.0 532 598 668 743 82.1 903 99.0 

estnpkis | 25.0 288 329 372 41.9 468 520 57.5 632 693 


= 2s = = sea 
B=5, ¢)B°=25, c7=.16, ¢y=.35, cg=.76 


average time for the slowest leaf router and the depth of the tree, d, are listed. 


santa Oe , a ei te 
Por coniparioun with the special uce sitmulauion results, we computed est7 4 and esinpkis g for 
various values of d. We assumed that C7 was equal to cg and that cg was equal to (.76) and we assumed 
that estnpkis 41 was equal to 25. The computed values are listed in Table [V. The simulation results 


seem to grow at Icast as fast as estT 7. 


We simulated complete indirect n-cube networks of modest size in order to compare their slow 
input routers to the slow leaf routers of special trees. Networks with bufters of size one, two, and five 
were simulated. [n the simulation, the outputs of the networks were connected to perfect sinks. A 
simulation run was made for cach combination of network depth and buffer size. For cach buffer size, B, 
a value of cy was selected. Each simulation run was divided into many pcriods such that cach input 
router accepted more that ¢ B? packets in cach period. For cach period of each simulation run, the time 
required by the slowest input router to accept a total of ¢} B 2 packets was measured. ‘The average over all 


the periods of a simulation run was computed. ‘The results of the simulation runs are shown in Table V. 


For comparison with the complete networks, we performed additional simulation of special trees. 


We simulated special trees with the same buffer sizes as the complete networks, We simulated a special 
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Table V. Evaluation of the Special Tree Model 


33.4 
24 


43 
214 


35.0 
.36 


58 
25.3 


42.4 
.29 


Pa 
25.9 


36.0 


29:5 


50.6 
25 


6 
30.3 


n=d+4= l 2 3 4 5 
= l 
B=1 
cBr=4 
Laeleiy 4.6 TA 13.1 15.1 22.6 
2 
cB MAT ei daw) 43 28 AS 13 09 
IR f2" 38 31 26 23 21 
T spslow 9.1 
(cy = .23) 
B=2 
¢B7=16 
Toa 10.5 14.4 19.1 27.8 39.3 
pe 
Cy BV QT stow) +76 56 42 35 .26 
(Ro A 14 62 54 49 46 
T spslow 16.1 
(cy =.23) 
B=5 
cB 2. 25 
Tea 28.7 31.7 
2 
cB 127 iow) 44 39 
Rafe 62 60 
T spslow 19.1 
(cy =.23) 
where L alow is the time required by the slowest input router of an indirect n-cube network 
to accept a total of cB 2 packets 
ER is the average total input rate of an indirect n-cube network 
and de is the time required by the slowest leaf router of a d-stage spccial 


spslow 
to accept a total of cB 2 packets 
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tree for each combination of depth and buffer size. For each special tree simulated. if d was the depth of 
the special then we set the parameters of the special to correspond to the parameters of a (d +4)-stage 
network with the same buffer size and with its outputs connected to perfect sinks. We did this in order to 
evaluate how well the d-stage special represented d-stage trees in the first d stages of a (d +4)-stage 
network as shown in Figure 21. As was discussed earlier, the behavior of a special tree is intended to 
represent the behavior of a tree in the first stages of a complete network. We chose to compare the 
d-stage special to d-stage trees of a (d +4)-stage complete network. While the exact choice of d +4 was 
not critical, it was important to consider a network that corresponded to the assumptions of the special 


trees. In particular, it was important to consider a network large cnough that the rate at which packets 


Fig. 21. d-Stage Tree ina (d + 4)-Stage Network 


d d+4 
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were accepted from the roots of the d-stage trees in the first / stages of the network was low. ‘The value 
of ey used for the special was equal to the value of cy used for the complete network. The value of cy, the 
parameter of the probabilistic sinks of the special, was selected to roughly correspond to the rate at which 
packets could be accepted by an input ofa router of the third from the last stage of the complete network. 
We estimated this rate by considering a four stage complete network with the same buffer size and with 
its outputs connected to perfect sinks, and by measuring the average rate at which its inputs could accept 
packets. A simulation run was made for each special tree. Each simulation run was divided into many 
periods such that if B was the buffer size, cach leaf router accepted more than cB? packets in cach 
period. For cach period of cach simulation run, the time required by the slowest leaf router to accept a 
total of cB 2 packets was measured. The average over all the periods of a simulation run was computed. 
The results are listed in Table V. The simulation results for the special trees do not scem to grow any 


faster than the simulation results for the corresponding complete networks. 


Table V also fists simulation results that can be used to compare the input rate of the slowest input 
router of an indirect n-cube network to the total input rate of the nctwork. For each simulation run, the 
table lists cB? divided by twice the average, over all of the periods of the run, of the time required by 
the slowest input router to accept ¢) B 2 packets, This quotient gives an indication of the rate at which the 
slowest input router accepts packets on each of its inputs. For each simulation run, the table also lists the 
average total input rate of the indirect n-cube network divided by the number of network inputs. These 
results suggest that the input rate of the slowest input of an indirect n-cube network can be several times 


slower than the expected input rate of a randomly selected input. 
Trees in Very Large Networks 


In very large indirect n-cube networks, n much larger than 10, the interaction between a large tree 
and the rest of the network docs not correspond to the assumptions that we made for modest trees in 


modest networks, n Tess than 9, Ina very large tree, as shown in Figure 22, conflict in the earlier stages of 
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the tree causes routers in the later stages to receive pa vets very slowly. Phe rate at which a router in one 
of the later stages of the tree receives packets on its inputs is not likely to be larger than the rate at which 
packets can be accepted from its outputs. The input buffers of such a router may often be empty. Asa 
result, the conclusions that we drew about the slow leaf routers of modest trees in modest networks do not 
hold for the slow leaf routers of very large trees in very large networks. Ford much larger than 10, we do 
not expect the slowest leaf router of a d-stage tree in an indirect n-cube network to require estT 4 time to 
accept cB? packets. We do not expect such a router to require time proportional to d2 to accept cy B? 


packets and estT F is proportional to d2, 


However, the time required by the slowest input router of a very large network to accept a constant 


number of packets is a function of the network’s depth. In particular, if the depth of the network is n, we 


Fig. 22. d-Stage Tree 


x 
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Cogn 
expect the slowest input router of the network to require greater than a n) ume to accept 3B packets 
2 


for some constant Cg. 


The motivation for this can be seen by considering the network and its operation. For the purpose 


of discussion, we define 
5 fgh 102 
K = B(logy n) (102) 
We consider the routers in the (logy K)-1 st stage from the network inputs. For the purpose of 
discussion, we refer to the (log, K)-1 st stage from the network inputs as the selected stage. ‘The nctwork 
inputs can be divided into (N/K) groups with K inputs in each group such that the set of routers of the 


selected stage that can receive packets from one group of inputs is disjoint from the set of routers of the 


selected stage that can receive packets from any other group of inputs. 


We consider the operation of the network after some randomly chosen point in time and argue that 


Cgn 
with high probability at least one of the inputs accepts less than cB? packets in a period ob may units 
2 


of time after the selected point. We define P to be the chance that the first 32 K packets to arrive on a 
group of inputs must pass on the same output of the same router of the selected stage. Thus, 
P= KAKA , (103) 
P>(UKpPk , (104) 


and 
Mos Daan 2008”) sa K 
—) area 
a yt eee 
pbk N3¢9 
We define P’ to be equal to the chance that, for at least one of the groups of inputs, the first 3B K packcts 


P= (105) 
received by that group are tagged for the same output of the same router of the sclected stage. Thus, 
P? = 1-(1-pyN/K | (106) 


Using the Taylor series for log (1-P), 
; 7 cals ih 
Bo ae IAN ig ool ED, (107) 
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poy ee (NZE DP (108) 
and 

P = eo? (109) 
where exp is equal to (N/K )?. Thus, 

exp = (N/K)P 


= (NI 39 PBR ayKy, (110) 


B (logy n) 
C 

[f cg is much less than 1/3 and N is large then there is a good chance that for at least one group of inputs, 

all of the first 3B K packets received by that group are tagged for the same output of the same router of 

the sclected stage. In such a case, since only B(2K-2) packets can be buffered between that group of 

inputs and the output of the router of the selected stage, the operation of the router of the selected stage 

affects that group of inputs. Since there are K inputs in the group and since only one packet can be 


transferred per unit time on the output of the router of the sclected stage, at least one of the inputs will 


accept less than 38 packets in a period of BK units of time. In other words, at least one of the inputs 
Con 


9 
(log, n) 


will accept less than 38 packets in a period of units of time. ‘hus, if c) 2 is greater than three 


Cgn 
then at least one of the inputs will accept less than ec, B 2 packets in a period fT nD units of time. 
2 


2.3 Networks for Systems with Localized Communication 


Many localized communication patterns can be supported with networks that are less complex than 
the uniform communication networks described above. There are, of course, a wide varicty of localized 
communication patterns. While we have not done extensive work on this topic, we describe in this 
section an obvious family of network structures that seem appropriate for some important localized 


communication patterns, 


In the technologies of this chapter, there is a large class of systems such that each system can be 


supported by a network that has a cost linear in its number of inputs. Since we are assuming in this 
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chapter that all wires have the same cost independent of their length. the factors affecting the cost ofa 
network are simply the number and complexity of network nodes, and the number of interconnecting 
wires. [f cach source of packets in a given system generates packets for a number of destinations that is 
less than ¢ where c is soine constant independent of the total number of sources then the communication 
requirements of the system can be supported by a network with a number of nodes and wires 
proportional to the number of inputs. Such a network can be constructed in the obvious way by 
associating a node with cach input and a node with each output, and connecting each input node to the 
output nodes that correspond to possible destinations for packets from that input. The total cost of such a 
network is independent of the identity of the output nodes that must be connected to a given input node. 


It is important to note that this will not be the case in the technologies of the next chapter. 


Circ liicad Cost network structure is ure glid suuciure. We consider a grid of two dimensions as 
shown in Figure 23, but grids of higher dimensions are also useful. Each node of the grid is connected to 
the nodes adjacent to it. Each node is also connected to a network input and a network output. Clearly, 
the cost of such a network is linear in the number of network inputs. Such a network can obviously be 
used in systems that support computations on grid structured data such that the computation on a given 


grid element involves only the adjacent grid elements. 


Another linear cost network structure is the tree structure. In such a structure, the nodes are 
connected in a tree as shown in Figure 24. The inputs and outputs of the network can be connected in at 
least two possible ways. One way is to connect each node of the network to a network input and a 
network output. Another is to connect only the leaf nodes to the network inputs and outputs. In the 
discussion below, we assume that each node of the network is connected to a network input and a 


network output. 


A tree network can be used to support applications, such as divide and conquer algorithms, that 


require hierarchical communication patterns. A tree network can simultancously support communication 
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lig. 23. Grid Structure 


between the PC’s in cach pair of adjacent leaf PC’s. Thus, it can support a high total bandwidth for such 
communication. But the network has half that bandwidth for supporting communication that must go 
through the PC’s of the second stage from the leaves. In general, if for some i the network has some 
bandwidth for packets that must go through the /th stage then it has half that bandwidth for packets that 
must go through the /+I]st stage. Thus, the tree network can be used to support some systems that 
require hierarchical communication. For example, the network can obviously be used in a tree structured 
system where each module is the root of a subtree of modules that it controls and where cach module 


requires the same communication bandwidth. 


Vig. 24. Tree Structure 
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3. Design of Routing Networks Considering the Cost of Wires 
3.1 Introduction 


In this chapter, we examine the changes that will probably occur in integrated circuit technology in 
the next five to ten years, and examine the design of routing networks assuming such changes. We refer 


to the resulting technology as very large scale integration (VLSI). 


In the next five to ten years, we expect several improvements in integrated circuit technology. We 
expect a reduction in the width of wires and the size of transistors. The minimum width of wires may 
shrink to roughly 1/(2)!/2 to 1/(2.4) of its present value. The area required at the end of this period to 
implement a circuit may be 1/2 to 1/6 of the area. required at present to implement the circuit. We 
expect the speed of on-chip circuits to increase to roughly two to tour times their present speed. We also 


expect that by the end of this period the use of multiple layers of metal will be common. 


While we expect features on a chip to become smaller and the complexity of on-chip circuits to 


increase, we do not expect the overall physical size of the chips to increase drastically in this period. 


The improvements in integrated circuit technology will allow a large number of nctwork nodes to 
be placed on a single chip. As a result, the wires interconnecting the nodes will be on-chip wires. Since 
the chip area required to implement an on-chip wire is proportional to its length, the length of wires in a 


network subsection will be an important factor in the chip area required to implement the subsection. 


However, it appears that for the next five to ten years it will still be possible to drive even very long 
wires quickly by choosing drivers of the appropriate size. ‘The capacitance to the substrate per square 
micron of metal will increase as the thickness of the oxide layers decreases. We expect that the width of 
wires will in general decrease and that the area of a given Iength of wire will decrease. ‘The combination 


of increasing capacitance per unit area and decreasing area may cause the capacitance of a given length of 
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wire to remain roughly constant. A long wire with a large capacitance can be driven quickly ifa large 
driver capable of driving a large amount of current is used. Presently, even a cross chip wire can be 
driven in time comparable to the delay of about ten logic stages by using a transistor whose arca is a few 
times that of a minimum size transistor. [tis not clear exactly how the area required to implement a high 
current transistor will change as technology changes. The current driving capacity of a transistor is 
inversely proportional to the resistance of its channel. The transistor channel resistance for a particular 
ratio of length to width may increase. But since the minimum channel Iength will decrease, the minimum 
area for a transistor with a certain current driving capacity may decrease. It scems likely that for the next 
five to ten years it will still be possible to drive a very long wire in reasonable time with a transistor that is 


quite small in comparison to the area of the wire. 


til Wiis Chapter, we first descrive a tiiudel of VLSI ihat we will faie: use iu study the area required to 
implement network structures in VLSI. We expect that the dominant component of the total area 
required to implement a network in VLSI will be the area required to implement its wires. The features 
of the wires of the model correspond to what we have assumed above will be the primary characteristics 


of wires in VLSI. The wires of the model require an area proportional to their length. They have no 


propagation time. A signal can be asserted on a wire of the model in unit time. 


We then examine in this model of technology the design of networks for uniform communication 
applications. We examine in the VILSI model the fundamental cost of a single chip network to support a 
certain level of performance for uniform communication applications. We examine a few structures that 
seem appropriate for implementing a single chip uniform communication network in VLSI. These 
structures include a crossbar stnucture, and an indirect n-cube structure. We discuss a technique for 


interconnecting single chip networks to form larger networks. 


We also briefly examine networks for localized communication applications. We examine a few 


example network structures and describe the communication patterns that they can support. 
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3.2 VLSI Model 


The model presented here is for the most part the model suggested by Thompson [25]. ‘The items 
implemented on the chip can be broken into two primary types. processing centers (PC's) and wires. All 
processing occurs at the PC's. Transmission of information among the PC’s is performed using the wires. 


All switching functions are performed by the PC’s. 


It should be noted that the concept of unit time that we use in the VLSI model is different from the 
concept of unit time that we used in the previous chapter. In the previous chapter, a unit of time was the 
time required to transfer a single packet on a link. In the VLSI model, a unit of time is the time required 


to transfer a single bit of information on a wire. 


Each wire interconnects some number of PC’s. A wire has unit width, and a signal (one bit of 
information) may be asserted on the entire length of a wire in unit time. The model characterizes a wire 
as a lumped capacitive and resistive load with a rise time but with no propagation time. The model allows 
multiple conacctions to a single wire. It should be noted that the area charged in the model for such 
connections may be too small. ‘The primary reason for this comes from the fact that the model assumes 
that ina VLSI implementation the total area required for all of the drivers of a wire is comparable to, or is 
less than, the arca of a wire. In the case of a wire with only one or tie drivers, the area required to 
implement the drivers would be quite small in comparison to the area of the wire. However, in the case 
of a wire with a number of drivers proportional to its length, it is likely that the total area required to 
implement the drivers would be at least of the same order as the area of the wire. Thus, this model can be 
used to obtain lower bounds on the area required to implement a circuit, but the areca requircd for wire 
drivers must be considered carefully in the actual implementation of any circuit requiring a large number 


of connections to a single wire. 
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Each PC (processing center) is connected to some number of wires. [n this model, we assume that a 
PC is square and that the area required to implementa PC is at least as great as ie where # is the number 
of wires connected to the center, ‘This area is of the same order as that required to construct a complete 
cross point switch for switching among the wires. [fa center does not need such a powerful switching 
capability, it should be decomposed into smaller centers. We also assume that information can not pass 


through a node in Iess than one unit of time. 


We assume that cach piece of input data is available on-chip from a specialized input PC and that 
each piece of output data need only be delivered to an on-chip output PC. We have chosen to separate 
this model from the problem of getting information into and out of achip. The input and output capacity 
of VLSI chips depends critically on the technology used to package the chips. It is not presently clear 
wiiicli iccuindiugics will Lé used as ie scale of integration increases. This topic deserves further study as 


the packaging technology advances. 
3.3 Networks for Systems with Uniform Communication 
3.3.1 Wire Cost 


In this subsection, we investigate the wire area required by single chip networks capable of high 
performance in systems with uniform communication. In particular, we obtain for 1 > f > 0a lower 
bound on the area required in the VLSI model by the wires of any single chip N-input N-output routing 
network capable of supporting an average throughput of {/N packets per unit time when its inputs are 
connected to the uniform communication model sources and its outputs are connected to the non 
blocking model receivers. For this study, each model source is assuincd to produce a new packet within 
one time unit of the network’s acceptance of the source’s previous packet. We assume that the label of 
cach packet produced by a model source is independently selected, and that all of the possible destination 


labels are equally likely. 
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Proposition. For L> # >0,Q(f N)) area is required in the VISI model to implement any single chip 
N-input N-output routing network capable of supporting an average throughput of (/N packets per unit 
time when its inputs are connected to the uniform communication model sources and its outputs are 


connected to the non blocking model receivers. 


Proof. To get the desired lower bound for routing networks, we use an approach similar to that used by 
Thompson [25] for the discrete Fourier transform and by Abelson [1] for multiplication. This approach 
uses a concept called the minimum bisection width, which we will define in terms of the graph of a VLSI 
circuit. For any circuit in our VLS{ model, the graph of the circuit is defined to be G = (V,E) where V 
contains a vertex for each of the PC’s in the circuit, and E is the set of all sets {x,y} such that x and y are 
contained in V and a wire exists between the two PC's corresponding to x and y. [he minimum bisection 
width of the circuit is defined to be the smallest 6 such that for some partition of V into Hy and H, with 
HI < [EL] < IHyI + 1, the deletion of & edges from E can disconnect Hy, from H5. The minimum 
bisection width of a subgraph is similarly defined. If U is a subset of V for some graph G = (V,B), then 
the minimal biscction width of U in G is defined to be the smallest 6 such that for some partition of U 
into H, and Hy with |Hy| < |H»| < |Hy| + 1, the deletion of b edges from E can disconnect Hy from 
Hp. Thompson has shown that if the minimum bisection width of some subset of the graph of a VLSI 
circuit is b, then the area required in the VLSI model for the circuit’s wires and PC’s is greater than b 274. 
Thus, if we can develop a lower bound on the minimum biscction width of any VLSI circuit capable of 
performing a particular function in a given period of time, we can deduce a lower bound on the area 


required by any such circuit. 


A lower bound on the minimum biscction width of any single chip VLSI implementation of any 
N-input N-output routing network with the desired characteristics can be established by examining the 
communication needs of such a network. Let us consider the graph, G = (V,E), of any such VISI 


implementation. We assume that the VISE implementation has N input PC’s and N output PC's. We 


assume that each input PC can produce packcts as fast as the network can accept them and cach output 
PC can accept puckets as fast as the network can present them. We will examine the case for even N. A 
similar approach can be used for odd N by ignoring one input PC and one output PC. Let O be the 
subset of V corresponding to the N output PC’s, and I be the subset of V corresponding to the N input 
PC's. If the minimum bisection width of O in G is &, then there must be a set of & edges whose removal 
would cause onc half of the vertices in O to be disconnected from the other half. Let O; and QO be the 
two disconnected subsets of O that would result from the bisection where |O;| = Oy]. Let 1, be the set 
of all vertices in [ that would remain connected to any vertex in O, after bisection, and 15 be the set of all 
vertices in I that would remain connected to any vertex in Op. ly) and ly must be disjoint since by 
definition the bisection disconnects O; and Oy. It follows from our previous assumption regarding the 
average throughput of the network, that in some very long period of duration T the network must be able 
to accept at least f NT packets. The characteristics of the model sources imply that the expected number 
of packets received during such a period that must be routed either from inputs in 1) to outputs in Oy or 
from inputs in [5 to outputs in Oy is greater than or equal to f N72. Since cach wire can transmit only 
one bit per unit time, there must be at least /N/2 wires corresponding to the edges in the biscction. 


Therefore 6, the number of edges in the minimal bisection of O in G, must be at least f N/2. 


Based on these results and Thompson’s theorem it follows that the area required in the VLSI model 
to implement any N-input N-output routing network with the capacity to support an average throughput 
of f N packets per unit time in the uniform communication model application is Q((fN)~). fn other 


words, there exists a constant c such that the area is greater than or equal to c(f/N). 
This ends the discussion of the proposition. # 


If we make certain additional assumptions about the network, we can obtain a more detailed lower 
bound on the area required to implement the network in the VLSI model. In particular, if we assume 


that there are at least p bits in cach packet, and if we continue to assume that the VLSI implementation 
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has N input PC’s and N output PC’s, then we can show that Q(p f'N)") area is required in the VISIT 


model to implement the network. The argument is similar to the one used above. 
Thus under these assumptions, p and N have the same effect on our lower bound. 


As we shall see later, some of the networks that we present come close to this bound. In particular, 
they can support a throughput of Q(N a aw packets per unit time and require O((w Ny?) area in the 
VILSI model where p is the number of bits in cach packet and w is the number of wires in cach link of the 
network. Thus, the area of the networks differs from the lower bound by a factor Sera This factor is 


a result of the fact that we assume that a nctwork node requires log w time to receive and acknowledge a 


group of w bits where one bit comes from each of the w wires ofa link. 


In considering these results, it should be remembered that the nature of the communication 
networks required for a particular system depends on the overall design of the system. Onc important 
issuc in the design of a large system is the decomposition of the system into subsystems such that each 
subsystem can be implemented on a single chip in VLSI. A number of factors affect the decomposition 
of the system. The nature of the modules to be interconnected by a network affects the chip arca 
required to implement them, and it affects the feasibility of implementing them on the same chip as the 
network. The maximum area of a chip and the maximum number of pins on a chip limit the complexity 


of and the communication bandwidth of a single chip subsystem. 


The decomposition of a system affects the characteristics of the communication networks required 
by the system and thus affects the cost of those networks. ‘lo illustrate this, we consider a system with two 
possible multiple chip implementations. We refer to these two implementations as implementation, and 
implementation. implementation, is composed of N single chip modules interconnected by w single chip 
uniform communication networks. In implementation j» we assume that cach of the single chip modules 


has w inputs of one wire cach and w outputs of one wire each, and we assume that cach of the networks 
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has N inputs of one wire each and N outputs of one wire cach. implementations is composed of N single 
chip modules interconnected by a single uniform communication network. [n imiplementationy, we 
assuine that cach of the single chip modules has one input of w wires and one output of w wires, and we 
assume that the network has N inputs of w wires each and N outputs of w wires each. The lower bounds 
above suggest that a N-input N-output uniform communication network with w-wire data paths requires 
more areca in the VLSI model than a N-input N-output uniform communication network with single wire 


2 times as much area as a network 


data paths. In particular, a network with w-wire data paths requires w 
with single wire data paths. Thus, the total area in the VLSI model required for the networks of 


implementation, is less than the area required for the network of implementation by a factor of w. 


In order to further demonstrate the importance of the decomposition of a system into single chip 
subsystenis, we consider a third iitipicrientaiivu uf the system of the previous paragraph. We refer to unis 
implementation as implementations. implementationg is similar to implementation, except that all of the 
components of implementation; are placed on the same chip. In particular, implementation, is a single 
chip composed of N modules interconnected by w uniform communication networks. — In 
implementation, each of the modulcs has w single wire inputs and w single wire outputs, and cach of the 
networks has N single wire inputs and N single wire outputs. In the VLSI model, it can be shown that 
implementation 3 requires area of the same order as the communication network of implementation. 
Thus, implementation, requires w times as much chip area as that required by the networks of 
implementation}. The discrepancy is due to the fact that some of the interconnections that are 
accomplished by on chip wires in implementation; are accomplished by off chip wires in implementation). 
In particular, the connections in implementation, between the modules and the networks require 
Q((Nw )) chip area but the corresponding connections in implementation, are accomplished by off chip 


wires. 
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Thus, we have shown that the data paths ofa single chip uniform communication network imply a 
certain fower bound on the area required by the network in the VISE model. In particular, O((p fN)*) 
area ts required in the VISTI model to implement any single chip N-input N-output routing network with 
an average throughput of (N p-bit packets per unit time in the uniform communication model 
application. However, in considering this lower bound it should be remembered that the decomposition 
of a system into single chip subsystems affects the characteristics of the networks required by the system 


and thus the cost of those networks. 
3.3.2 Network Structures 
Introduction 


A number of structures exist tor single chip routing networks that are capable of high pertormance 
for uniform communication applications. These include a simple structure similar to ihe standard 
crossbar switch as well as the indirect n-cube structure. While the overall areas required by these 
structures in the VLSI model are similar, many of the other characteristics of these structures differ 
greatly. For cxample, the N-input crossbar nctwork uses wires with N connections, but the N-input 
indirect n-cube network uses only wires with two connections. We examine a few network structures and 


examine the characteristics of each structure that affect its VLSI implementation. 


We assume that w-bit wide data paths are used for cach network structure. For each structure, as 
we Shall see, the w width of the data paths results in a w2 factor in the arca required to implement the 
structure. The arca required for cach of the network structures is O((w N)?) where w is the number of 


wires in each link. 


Thus for cach of these structures, much less area is required in the VST model for w networks with 


single wire data paths than fora single network with w-wire data paths. 


i % = 


Crossbar Structure 


‘The first network structure that we examine is similar to the standard crossbar switch. We describe 
the structure, discuss its complexity, and discuss two of its drawbacks. ‘hese drawbacks are the need for 


many drivers for cach long bus and the need for bus arbitration, 


A N-input N-output network built according to this structure is composed of N2 PC’s arranged ina 
grid with an additional N input PC’s and N output PC’s as shown in Figure 25. Each of the PC’s in a 
given row is connected to w wires associated with that row. Similarly, cach of the PC’s in a given column 
is connected to w wires associated with that column. Each row of the grid is associated with one of the 


network inputs. Each column of the grid is associated with one of the nctwork outputs. The input PC’s 


Fig. 25. Crossbar Network 
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are connected to the network inputs, and the output PC’s are connected to the network outputs. A packet 
entering the network is initially stored in a input PC. The destination label and data for one packet are 
transmitted by the input node across one row of the grid. Each PC in the grid is capable of determining if 
its associated output column is idle and if its associated input row is presenting data for that output 
column. In such a case, the grid PC connects the input row to the output column, and the output PC 
copies the presented data. Once the output PC has safely stored the packet, the grid PC will terminate its 
connection, and the input PC will present its next packet. Each output PC passes the packets it has 


received over the nctwork output associated with it. 


Obviously, (N2 + 2N) PC’s are required for a N-input N-output network built according to this 
structure. Further, if the area required for each PC is independent of the size of the nctwork, then the 
overall aictwork layout can be done in thie prid like fashion we have described using a toial area which is 


proportional to (Nw) where w is the number of wires in each data path. 


There are some problems associated with the implementation of very large networks that have this 
structure. The first comes from the fact that cach output column wire can be driven by any of the N PC’s 
in the column. For the reasons that we discussed earlier, the total arca required for the drivers of a 
column wire in an actual implementation may be larger than the area of the wire. The area required to 
implement a N-input N-output crossbar network may grow faster than N2. However in the technology of 


the next five to ten years, we expect that the growth will be close to N2. 


In addition, there are problems asscciated with the control of the various grid PC’s. Only one grid 
PC in a given column should be allowed to drive the column at a given time. Onc way to accomplish this 
is to view the column as a synchronous bus and to use a grant signal that is daisy chained through the 
PC’s of the column. Unfortunatcly, there are problems with this scheme. This scheme requires that a 
clock signal be distributed to all the PC’s of the column. This scheme can only implement a fixed priority 


of inputs for the column. Further, the grant signal of this scheme is quite slow since it has to go through 
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the control circuitry of each of the N+ 1 PC's of the column. In the VEST model, Q(N) time is required 
for the grant signal to reach the lowest priority input row. ‘This time would seem to be a serious problem 
since we would like each input row of the network to be capable of transmitting packets at a high rate. 
However in a real implementation, the speed of the control circuits in cach PC may be much greater than 
the speed of an input row driver. For example, the grant signal may be propagated through a PC in one 
tenth of the time required for a signal to be asserted on an input row wire. For networks of the size that 
we expect in the next five years, networks with perhaps 64 to 128 bit serial inputs, the time required to 
chain a grant signal through all the PC's of a column may be comparable to the time required to send bit 
serially the destination address of a packet along an input row. In such a casc, it may be feasible to 


implement a crossbar network with output columns with daisy chained control wires. 


Anoitlicr possible techuique for gbiaining ine situiual exclusion among de gud PC's conuccied to a4 
column uses a N-input tree of arbitration units (Figure 26) for cach column. Each arbitration unit has 


two incoming request lines from the two arbitration units below it and one incoming grant line from the 


Fig. 26. Arbitration Tree 
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arbitration unit above it, When the arbitration unit receives a request on either or both of its incoming 
request lines, it will produce a request on its outgoing request line. When the arbitration unit receives a 
grant, cither one or both of the arbitration units below it must have a pending request. If there is only 
one pending request, then the unit returns a grant for that request. Otherwise, the unit arbitrarily chooses 
one of the requests and returns a grant for that request. A grant is removed only after its associated 
request has been removed. ‘The N-input arbitration tree associated with a given column has an incoming 
request line from and an outgoing grant linc to each of the PC’s in that column. The arbitration tree 
ensures that only one PC in a column can reccive a grant at a given time. Unfortunately in the VISTI 
model, Q(log N) time is required for the request from a PC to receive a grant from the arbitration tree. 
Since we would like cach input row of the network to be capable of transmitting packcts at a high rate, 
the delay of the arbitration tree would scem to be a problem. However in a real implementation, the 
speed of an arbitration unit may be much greater than the speed of an output column driver, and the total 
delay of an arbitration tree may be comparable to the delay of an output column driver. It should be 
noted that if the N PC’s of acolumn are in a straight line, the arbitration tree for the column may require 
area proportional to N(log N). The best layout that we know for a crossbar network with arbitration trees 


requires Q(N2 w (log N) + N2 w2) area in the VLSI model. 


Other techniques exist for maintaining mutual exclusion on the output columns. For example, each 
column wire can be viewed as a broadcast medium, and the mechanisms that have been developed for 
conflict resolution in broadcast networks can be applicd. Mechanisms of this sort have been studied 
extensively in the literature [18]. Unfortunately, these mechanisms secm to require complex strategies for 
determining when a message should be retransmitted after a collision. ‘Thus, these mechanisms seem to 


require a rather complex input PC for each input row. 


The performance of a crossbar network depends, among other things, on the technique used for 


obtaining mutual exclusion on the output links. For uniform patterns of communication the crossbar 
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N 
log w) + (arbitration time) 


network can support a throughput of Q¢ ) packets per unit ime where w is 


as 
the number of wires in cach link and p is the number of bits in each packet. ‘The denominator 
corresponds to the time required to handle a single packet, which is the time required to gain control of 


the necessary output column plus the time required to transfer the packet. We assume that log w time is 


required to transfer w bits through a PC where one bit comes from each of the w wires of a link. 


Indirect n-Cube Structure 


The indirect n-cube (InC) structure, which we described in the last chapter, can also be used to 
construct a single chip routing network capable of achieving high throughput in applications with 
uniform communication. This structure has a number of characteristics that make it interesting for VLSL 
Yhe inC network requires only two connections to each wire and thus avoids most of the implementation 
problems associated with the crossbar network. As we shail sce below, the arca required in the VUST 
model to construct a N-input N-output InC routing network is O((N w y*) where w is the number of wires 


in cach data path. 


Onc possible approach for laying out a N-input N-output InC network with w width data paths in 
O(Nw )) area is shown in Figure 27. It should be noted that the figure shows the case for w equal to 
one. Layouts for w not equal to one can be obtained by replacing cach wire in the figure with a group of 
w wires. A N-input network is constructed from two (N/2)-input networks and N/2 two-input routers. 
We assume that at least two layers exist and thus crossovers are possible. The first output of the first 
router is connected to the first input of the first component (N/2)-input network. The last output of the 
last router is connected to the last input of the second component (N/2)-input network. Other 
connections between router outputs and inputs to the component routing networks are accomplished 
using (N-2)w vertical wires. In particular, the jth set of w vertical wires connects the second output of 


the (i +1)/2 router to the (¢ +1)/2 input of the second component network if / is odd and connects the 
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Fig. 27. Layout for an nC Network 
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first output of the (7 +2)/2 router to the (i +2)/2 input of the first component network if / is even. 
Thus, the areca required to implement a N-input network, A(N), is less than 2A(N/2) + cy (Nw)? for 


some constant ¢). This recurrence implies that A(N) is less than 


2ey(Nw)? + cyN (111) 
for some constant ¢>. This can be verified by substituting 2c,(Nw)? + ¢,N for ACN) into the inequality 

A(N) < 2A(N/2) + c(Nw)? (112) 
We act 

2e(Nw)? + cyN 2 4e)(Nw/2)? + 2e9(N/2) + ¢(Nw)? (113) 
or 


2ey(Nw)? + cyN = 2cy(Nw)? + oN (114) 


- 102 - 


While this layout is certainly not the smallest possible, it does demonstrate that the InC network can be 


implemented in O((Nw )?) area. 


{t should be noted that there is an imbalance between the wires and the PC’s of the [nC network. 
In the VLSI model, the InC network requires only (N/2)(logy N) PC’s but it requires O((Nw)2) area for 
its wires. In addition, the PC’s of the InC network are complex. Thus, it seems that most layouts for the 


InC network will not be homogencous but will instead contain separate areas for wires and PC’s. 


The throughput of indirect n-cube networks for uniform patterns of communication was discussed 
in the previous chapter. The strongest constraint that we studied in the previous chapter still allows a 
throughput of Q(N) with the model of time of the previous chapter where N is the number of inputs. In 
the VLSI model, this would suggest a throughput of verre packets per unit time where p is the 


factor comes from the fact 


number of bits in a packet and w is the number of wircs in a link. ‘The = 
that we assume (log w) time is required for a PC to process w bits where one bit comes from each of the 


w wires of a link. 


While N-input InC networks with w-wire data paths and N-input crossbar networks with w-wire 
data paths both require O(Nw)2) area in the VLSI model, there are some important differences between 
the two networks. The N-input [nC network has only (N/2){logy N) PC's, but the crossbar has (N2+2N) 
PC’s. The nodes of the InC network are more complex than the nodes of the crossbar network. Each 
node of the InC network requires a buffer on each of its inputs and requires a control circuit that is more 
complex than the control circuit of a node of the crossbar network. As a result, more arca is required in 
the VI-SI model to implement a node for the InC network than to implement a node for the crossbar 


network. 
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Other Network Structures 


There may well exist structures that are better suited for single chip high performance uniform 
communication networks than the [nC structure and the crossbar structure. We have discussed the areas 
required in thi VISI model by the crossbar network and the [nC network. There are problems with both 
the VLSI implementation of the crossbar network and the VLSI implementation of the InC network, As 
we have discussed earlier, each of the column wires of the crossbar network must be driven by a large 


number of PC’s. ‘The [nC network has an imbalance between its wires and its PC’s. 


We have examined two other network structures. For N-input N-output networks, these structures 
require Q(N2) PC’s and have two PC’s connected to each wire. The restriction on the number of 
connections to a wire ensures that these structures do not have the problems associated with 
implementation of multiple drivers for a single wire. Further, these structures require roughly the same 


total area for PC’s as they require for wires, and these structures have simple regular layouts. 


One network structure that we have examincd is the forest network. The N-input forest network is 
composed of 2N large trees. N of these trees are N-output switch trees, and the other N trees are N-input 
merge trees. A N-output switch tree is constructed from two (N/2)-output switch trees and a switch as 
shown in Figure 28. A switch is a device with one input and two outputs, and has the capacity to buffer 
some number of packets. The switch routes a packet according to the packct’s destination tag. A 
two-output switch tree is simply a switch. Similarly, a N-input merge tree is constructed from two 
(N/2)-input merge trecs and a merge as shown in Figure 29. A merge has two inputs and one output and 
some amount of internal buffering. The merge funnels all packets on its inputs to its outputs. Packets are 
output in the order in which they are accepted, and inputs are examined in a round robin fashion. A 


two-input merge tree is simply a merge. 
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Vig. 28. Switch Tree 
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Fig. 29. Merge Tree 
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Each input of a N-input N-output forest network is connected to the input of a separate N-output 


switch tree, and each output of the forest nctwork is connected to the output of a N-input merge tree. For 
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cach ¢ and / such that] <i < Nand 1 <7 <N, the sth output of the switch tree associated with the 
ith network input is connected to the /th input of the merge tree associated with the jth network output. 
Thus, each switch tree sorts the packets from a given input according to their destination addresses, and 


each merge tree collects all packets destined for a given output. 


Unfortunately, we have not found a good layout for the forest network. We have found a simple 
layout, Figures 30-32, that requires O((Nw log N)?) area in the VLSI model. We have not shown that 
this layout is the smallest possible. It should be noted that the figures show the case for w equal to one. 
Layouts for w not equal to one can be obtained by replacing cach wire in the figures with a group of w 


wires. 


We have studied another network that is related to the crossbar network. We call this network the 
checkerboard network. The checkerboard network, shown in Figure 33, has a grid structure much like 


that of the crossbar network with a row corresponding to cach input and a column corresponding to each 


Fig. 30. N Input N Output Forest Network 
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Fig. 31. 0,n Forest Subsection 


wires 


output. But unlike the crossbar network that uses one set of w wires to connect the PC’s in a given row, 
the N-input checkerboard network uses N sets of w wires. (N-1) sets of w wires connect adjacent grid 
PC’s in the row, and one sect of w wires connects the input PC to the nearest grid PC in the row. 
Similarly, the N-input checkerboard network uses N scts of w wires to connect the PC’s in a given 
column. (N-1) sets connect adjacent grid PC’s in the column, and one set connects the output PC to the 
nearest grid PC in the column. Thus, each wire in the checkerboard network is connected to only two 


PC's. 


The grid PC’s of the checkerboard network, unlike the grid PC’s of the crossbar network, have the 
capacity to buffer some number of packets. A packet is transferred from an input to an output by passing 


it along a path of connected PC’s between the two, 


‘The N-input checkerboard network, like the N-input crossbar network, requires (N2 + 2N) PC's, 


and can be laid out using a total area which is proportional to (Nw), 
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Vig. 32. gn Forest Subsection (i < n) 
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Although the throughput of the checkerboard network in uniform communication applications may 
be very good, the time required for each packet to be transmitted through the network is very long. In 


particular, if the inputs of a N-input checkerboard network are connected to model uniform 
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Vig. 33. Checkerboard Network 
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communication sources, then the expected number of PC’s in the path through the network taken by a 
randomly arriving packet must be Q(N). ‘This implies that the average delay for a packet is Q(N (log w)) 


units of time. This long average delay time is the primary weakness of the checkerboard network. 


In the technology of the next five to ten years, the checkerboard network seems less interesting than 
the crossbar network. In this technology, it should be possible to build single chip crossbar networks and 
single chip checkerboard networks of approximately the same size. ‘The two networks are capable of 
similar throughput for uniform communication applications, but the expected delay through the 
checkerboard network is much greater than the expected delay of the crossbar network. However, it is 


difficult to predict the changes that will occur in technology in the more distant future. Networks such as 
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the checkerboard network may become more important as chip complexity increases and the feasibility of 


the multiply driven wires of the crossbar network decreases, 
3.3.3 Multiple Chip Networks 


If the network required by a particular system can not be implemented on a single chip then, 
obviously, the network must be implemented as an interconnection of several chips. One technique for 
constructing a large composite network involves the interconnection of several single chip networks. For 
the purpose of discussion, we refer to the component single chip networks of such a composite network as 
the subnetworks of that network. The subnetworks of a composite network are mounted on circuit 
boards and are interconnected by board wires. Thus, the issues involved in the interconnection of the 
subnetworks of a composite network are similar to the issucs involved in the interconnection of the nodes 
of a network of the previous chapter. [In particular, the length of the wires used to interconnect the 
subnetworks has less effect on the overall cost of the composite network than the number of such wires 
and the number of subnetworks. Thus, interconnection patterns similar to those used in the previous 
chapter may be appropriate for interconnecting the subnetworks of a composite network. In particular, 
an interconnection similar to the indirect n-cube structure seems interesting. This interconnection has the 
form shown in Figure 34. If a@-input a-output subnetworks are used then the composite network 
contains log q N Stages of subnetworks. For the purpose of discussion, we number the stages from the 
network inputs to the network outputs with the stage connected to the network inputs being the zeroth 
stage. The th stage is divided into a i groups of subnetworks. Each group of the ith stage has @ 
associated groups in the (7 + 1)st stage. ‘The jth output of each subnetwork of a group of the /th stage is 


connected to the jth associated group of the (/ + 1)st stage. 


If a composite network is constructed from single chip indirect n-cube networks in the manner 
described in the previous paragraph then the composite network is an indirect n-cube network. Ifa 


composite network is constructed from single chip crossbar networks in the manner described in the 
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Fig. 34. Nx N Composite Network 
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previous paragraph then we expect the average throughput of the composite network to be at least as 


large as that of an indirect n-cube network of the same size. 


ol A 


3.4 Networks for Systems with Localized Communication 


Many localized communication patterns can be supported by networks in VISIT that are 
substantially cheaper than a uniform communication network. However, the design of networks in VLSI 
for localized patterns of communication must take careful consideration of the cost of wires. In the 
technologies of the previous chapter, we determine the cost of a network by considering the number of 
wires, and the number and size of nodes of the network. In VELSI, the length of cach wire must also be 
considered. If each source module of a system generates packets for only a constant number of 
destinations then the communication requirements of the system can be supported in the technologics of 
the previous chapter by a lincar cost network. However, many such systems require networks with 
greater than linear cost in VLSI. For example, the "perfect shuffle" pattern [24] can be implemented 
using a number of wires proportional! to N but it requires Q(N/log N)?) wire area in the YLUS! mode 


[13]. 


In this section, we describe two obvious but important networks that can be implemented in VLSI 
in area proportional to their number of inputs. While we do not examine these networks in great detail, 


we do discuss some of the issues involved in their VLSI implementation. 


The first of these networks is the grid network. We consider two dimensional grids as shown in 
Figure 35. Each grid PC is connected to the grid PC’s adjacent to it. If each grid PC is connected to a 
network input and a network output then the number of PC's is obviously lincar in the number of 
network inputs and outputs. The area required to implement the wires that interconnect a given PC to 
PC’s adjacent to it is proportional to the area of the PC. Thus, the total area required for the wires of a 


grid nctwork is lincar in the number of network inputs. 


In VLSE one of the biggest issues in the implementation of linear cost networks in general and grid 


structured networks in particular may be the constraint placed by the limited number of input and output 
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Vig. 35. Grid Network 


connections to a chip. Presently, chips with 64 connections are commonly available. In the next five to 
ten years, packaging technologies capable of handling chips with 100 to 200 connections should be 
common. However, it will probably be possible to implement a grid network with several thousand PC’s 
onasingle chip. This suggests that ifa system of modules interconnected by a grid structured nctwork is 
to be implemented in VLSI, it may be better to place some of the modules and a portion of the network 
on each chip than to place the modules and the network on separate chips. If some number of the 
modules and the portion of the grid network required to interconnect them are placed on a chip then the 
input and output requirements of the chip may be modest. The only signal wires that need to go off of 
the chip are those wires that connect the grid of the chip to the grids of other chips. ‘These wires connect 


to the PC’s along the perimeter of the grid of the chip. 
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The second network that we consider is the tree network, fn such a network, the PC’s of the 
network are connected ina tree as shown in Figure 36. The inputs and outputs of the network can be 
connected in at least two possible ways, One way is to connect each PC of the network to a network input 


and a network output. Another is to connect only the leaf PC’s to the network inputs and outputs. 


A tree network can be implemented in a small area in VLSI. A tree network can be laid out in 
O(N) areca as shown in Figure 37. But there are some problems caused by the limited number of 
connections that can be made to a chip. As was the case for the grid network, it may be possible in VISTI 
to implement on a single chip a tree network with a large number of PC’s but such an implementation 
could not have an off chip connection for each PC. This suggests that cach chip in the VLSI 
implementation of a large tree structured system should contain both a portion of the network and the 
alte iO GE Cotuiceivd lo Wat portivg of the uviwotk, Uulike die case fur a gitd uciwotk where the 


perimeter PC’s represent only a smail fraction of the PC’s of the network, the leaf PC's of a tree network 


Fig. 36. Tree Network 
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rig. 37. Layout for a Tree with N Leaves 
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represent over half of the PC’s of that network. Thus, while it may be possible on a single chip to 
implement a tree structured subsystem with a large number of very small modules, it is probably not 
possible to make an off chip connection for each leaf module of such a subsystem. As a result, it may be 
difficult to decompose a large tree structured system of very small modules into subsystems such that 
each subsystem requires the area of one hip and such that no subsystem requires more connections than 
the number of connections that can be made to a single chip. However, in the technology of the next five 
to ten years this may be a problem only for systems with very small modules. Unless the modules of a 
system are very small, it is likely that only a few dozen of the modules fit on a single chip and it should be 


possible to provide an off chip connection for cach module of cach chip. 
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4. Conclusion 
4.1 Summary 


In this thesis we examined the design of routing networks under two different sets of assumptions 
about the implementation technology. One set corresponds to present (1984) technology where only a 
small number of nodes can be implemented on a single integrated circuit. The other set corresponds to a 
VLSI technology where a large number of network nodes can be implemented on a single integrated 


circuit. 


In present technology, we examined the design of routing networks for systems with uniform 
communication and we briefly examined the design of routing networks for systems with a few particular 


patterns or localized communication. 


We showed that Q(N log N) nodes are required by any N-input N-output network capable of 
supporting an average throughput of Q(N) packets per unit time for our model of uniform 


communication. 


We studicd in detail one particular routing network, the indirect n-cube routing network, which 
seems well suited for uniform communication and requires O(N log N) nodes. We examined certain 
important characteristics of the operation of very large indirect n-cube networks and the effect of these 


characteristics on network performance. 


We examined the buffering of packets in front of a slow router. Such buffering involves a tree of 
routers in front of the slow router. Our model suggests that expected number of packets buffered in front 
IN 
= =e. 
of the slow router is greater than 2 2OUT-IN KB +1) Z 1 where /N is the rate at which packets are 


generated on cach network input, 8 is the size of the buffer on cach input of each network node, and 


OUT is the rate at which the slow router can accept packets. 
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We examined the effect that congestion in routers ofa given stage of an indireet n-cube network has 
on the network's performance. We chose to study the effect of congestion in routers of the last stage since 
analysis of the last stage is somewhat easier than analysis of other stages. We examined the buffering 
caused by such congestion and the effect of such buffering on network throughput. Our study suggests 
that congestion in a single stage of routers does not place a severe constraint on the throughput of the 
network. Our study suggests that this type of congestion still allows the normalized throughput of the 
network (the total throughput of the network divided by the number of network inputs) to approach a 
non zero constant as the size of the network goes to infinity, and that even for modest buffer size this 


constant is not significantly less than the normalized throughput of a two-input two-output network. 


We examined the effect of the interaction of routers of different stages on network performance. 
We Studicd tie iiteraction of routers aluny a uciwork path and we siudicd the uleracuun of touiers un a 


tree of the network. 


We studied the interaction of routers along a network path primarily by simulating a model of a 
network path. Our model reflects the interaction of routers along a randomly sclected network path while 
ignoring the interaction between a router on the path and any router not on the path. Our study suggests 
a limit on the input rate of a randomly selected network path and thus implics a fimit on the overall 
throughput of the network. This limit on network throughput is stronger than any of the other limits 
studied. However, this limit still allows normalized throughput to approach a non zcro constant as 


network size approaches infinity. 


We examined the interaction of routers in a tree of the network. We examined onc particular type 
of interaction that occurs in trees of modest size, trees of less than 10 stages. Our study indicates that this 
type of interaction does not have an important effect on the overall throughput of the network but it docs 
cause a few of the routers connected to the network inputs to be slow for a long period of time. For 


example, our study suggests that this type of interaction can cause the input rate for the slowest input of a 
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network with cight or Hine stages to be less than hal! the expected input rate for a randomly selected 


input for a period of forty units of time. 


We also bricfly considered a factor that has an influence on the speed of the slow inputs of very 
large networks. ‘This factor causes the slowest input router to require Cen time to accept 32 packets 


where # is the buffer size and n is the depth of the network. 


In summary, our work indicates that for present technology the indirect n-cube network is a good 
network for handling uniform communication. ‘he strongest constraint on throughput that we studied 
sull allows throughput to grow linearly with network size. However, our study also indicates that even in 


indirect n-cube networks of modest size some of the network inputs can be slow for a long period of time. 


We briefly examined one obvious family of networks that are appropriate in present technology for 
some important localized communication patterns. This family includes grid structured networks and 


tree structured networks. 


We also bricfly examined the design of routing networks in VLSI. We described a model of VLSI 
based on assumptions about the characteristics of VLSI. The model reflects the fact that in VLSI the cost 


ofa wire is proportional to its length. 


We examined the design of uniform communication networks in VLSI We showed that Q((f N)?) 
arca is required in the VLSI modcl to implement any single chip N-input N-output routing network 
capable of supporting an average throughput of fN packets per unit time for our model of uniform 
communication. We examined a few structures that are appropriate for unplementing a single chip 
uniform communication network. These included a crossbar structure and an indirect n-cube structure. 
The crossbar network is probably the most attractive since it has a simple regular layout. However, the 


crossbar network requires long buses with a large number of drivers that must be arbitrated. ‘The indirect 
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n-cube network does not require multiply driven buses, but it does require more complex network nodes 
and may require a more complex layout. We discussed a technique for interconnecting such single chip 


networks to form larger uniform communication networks, 


We briefly examined the design in VLSI of networks for localized patterns of communication. We 
discussed the fact that some networks that can be implemented in present technology with a number of 
nodes proportional to their number of inputs require greater than linear wire cost in the VEST modcl. We 
discussed two obvious but important networks that can be implemented in VLSI in arca proportional to 
their number of inputs, the grid network and the tree network. We discussed the pin out problems of 
both networks for very dense VLSI. We concluded that in very dense VLSI the processing modules of a 


system using cither network should be placed on the same chips as the modules of the network. 
4.2 Suggestions for Further Work 


There are many areas where further work could be done. Some of these are discussed below. 


There are interesting open questions concerning the performance of large indirect n-cube networks 
for uniform communication. The strongest constraint that we studied still allows the throughput of an 
indirect n-cube network for our model of uniform communication to grow linearly with the size of the 
network. However, we did not prove such a linear growth. Such a proof appears to be difficult. [t may 
be possible to obtain a proof if additional constraints are placed on the operation of the network. As was 


discussed in 2.2.3.1, an approach similar to Pippenger’s may be cffective. 


Clearly, more work can be done on the design of routing networks in present technology for 
localized patterns of communication. There are of course a wide variety of localized patterns of 
communication and it is probably not useful to try to examine all possible patterns. However, there may 


be interesting familics of communication patterns that can be efficiently supported by families of 
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networks. For example, there may be an interesting family of communication patterns What can be 
supported by the family of networks described below. Each network of the family corresponds to a tree 
network of the same size. As is shown in Figure 38, the network has ¢ links for each link of the tree 
network and a 3c x 3¢ subnetwork for each node of the tree network where ¢ is a constant. The 
subnetwork is some type of 3c -input 3¢-output routing network. Different members of the family have 
different values of c. A network of this family may be able to handle ¢ times as much traffic between 
distant nodes as the corresponding tree network. Other families of networks related to the tree network 
may be able to support interesting families of communication patterns. For example, [ciserson [17] has 
studied a more sophisticated family of networks called fat trees that secms to be able to support a wide 


class of communication patterns. 


wry 


Similaiy, more work can be done un ite design of singie chip routing ueiwurks ui ViSL. Tt would 


seem that low level implementation issues will continue for some time to be important in the design of 


Fig. 38. N ctree 
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single chip routing networks, ‘Thus, a more detailed took at the implementation of crossbar networks, 
indirect n-cube networks, and other potential networks is needed. In order to fairly evaluate data path 
sizes for the crossbar network and the indirect n-cube network, and arbitration schemes for the crossbar 


network, it may be useful to examine tentative chip layouts. 


One important issue that has not been considered in this thesis is the issue of real time fault 
detection and fault masking in routing networks. Some related work has been done elsewhere [2, 19], but 
more is needed. Detection of some faults can be accomplished by schemes that use check fields in each 
packet. Some types of fault masking can be accomplished if multiple paths exist between each source and 
cach destination. Such paths can easily be introduced in a network such as the indirect n-cube network 


by adding one or more additional stages. 


10. 


ll. 


Pe 


13. 
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